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PREFACE 


The  National  Science  Board  is  presently  exploring  how  the  U.S.  federal  government  sets  priorities  in 
research  and  development  and  whether  changes  are  needed  in  Ae  decision-making  process.  Accordingly,  Ae  NSB’s 
Committee  on  Strategic  Science  and  Engineering  Policy  Issues  asked  RAND’s  Science  and  Technology  Policy 
tostitute  for  a  comprehensive  review  of  Ae  relevant  literature  and  experience  on  R&D  priority  setting  across  fields 
of  science.  The  resulting  report  surveys  Ae  literature  to  identify  descriptions  of  Ae  budget  coordination  and  priority 
setting  meAodologies  currently  employed  by  Ae  federal  government  as  well  as  to  examine  critiques  of  currently 
employed  meAodologies.  The  report  will  be  of  mterest  to  Aose  wiA  general  mterest  m  Ae  realm  of  science  and 
technology  policy  and  specifically  treats  issues  of  priority  setting  and  coordmation  of  Ae  federal  R&D  portfolio 
across  fields  of  science. 

Created  in  Congress  in  1991,  Ae  Critical  Technologies  Institute  was  renamed  in  1998  as  Ae  Science  and 
Technology  Policy  Institute.  The  Institute  is  a  federally  fimded  research  and  development  center  sponsored  by  Ae 
National  Science  Foundation  and  managed  by  RAND.  The  Institute’s  mission  is  to  help  improve  public  policy  by 
conducting  objective,  independent  research  and  analysis  on  policy  issues  Aat  involve  science  and  technology.  To 
Ais  end,  Ae  Astitute 

•  supports  Ae  Office  of  Science  and  Technology  Policy  and  oAer  Executive  Branch  agencies,  offices, 
and  councils 

•  helps  science  and  technology  decisionmakers  understand  Ae  likely  consequences  of  Aeir  decisions  and 
choose  among  alternative  policies 

•  improves  imderstandmg  m  boA  Ae  public  and  private  sectors  of  Ae  ways  m  which  science  and 
technology  can  better  serve  national  objectives. 

S&T  Policy  Ttigtitiitfi  research  focuses  on  problems  of  science  and  technology  policy  Aat  involve  multiple 
agencies.  A  carry Ag  out  its  mission,  Ae  AstiAte  consults  broadly  wiA  representatives  fi'om  private  mdustry, 
mstitutions  of  higher  education,  and  oAer  nonprofit  mstiAtions. 

Aquiries  regardmg  Ae  S&T  Policy  InstiAte  or  Ais  document  may  be  directed  to: 

BAce  W.  Don 

Director,  Science  and  Technology  Policy  AstiAte 

1333  H  Street  NW 

Washmgton,  D.C.  20005 

Phone:  202-296-5000,  ext.  5351 

Web:  http://www.rand.org/centers/stpi 

Email:  stpi@rand.org 
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SUMMARY 


The  National  Science  Board  Committee  on  Strategic  Science  and  Engineering  Policy  Issues  asked  RAND  to  provide  a 
comprehensive  review  of  recent  literature  and  data  sources  on  priority  setting  and  coordination  in  federal  R&D.  The  foil 
review  presents  a  synthesis  of  how  the  literature  describes  priority  setting  across  fields  of  science  and  the  issues  involved. 

We  have  identified  gaps  in  the  literature  where  the  process  remains  unclear  and  needs  explication.  We  conclude  with 
suggestions  for  further  study.  The  following  summary  presents  a  cursory  overview  of  the  main  points  in  the  review. 

OVERVIEW  OF  FINDINGS 

General  Assessment  of  the  Literature 

•  The  literature  is  weighted  toward  the  prescriptive  rather  than  the  descriptive  and  tends  to  take  a  broad  view 
rather  than  examine  operations  at  the  agency  level. 

•  There  is  a  robust  literature  offering  advice  to  government  on  how  best  to  set  goals  and  allocate  funds,  bofli  as  a 
national  endeavor  and  across  governmental  agencies,  falling  into  fliree  broad  categories: 

•  a  shift  to  a  national-goals  approach,  tying  the  priority  setting  process  to  national  goals; 

•  a  scientific-goals  qjproach  advocating  cross-cutting  assessments  of  existing  spending  in  areas  of  science  and 
realignment  of  budgets,  if  needed,  to  further  scientific  advancement; 

•  fine-tuning  of  the  existing  complex,  political  process. 

•  A  smaller  but  growing  base  of  procedural  publications  describes  how  the  process  of  R&D  allocation  should  be 
done  widiin  an  agency  or  a  discipline. 

•  Only  a  few  reports  describe  how  the  process  actually  takes  place  within  the  government  and  no  publications 
describe  the  process  across  fields  of  science. 

•  There  is  only  a  sparse  literature  describing  efforts  at  coordination. 

•  There  is  a  richer  discussion  of  goals  and  priority-setting  within  the  Executive  branch  tiian  within  Congress. 
Qualitative  discussions  of  how,  or  even  whether.  Congress  decides  among  funding  options  for  different  areas  of 
science,  different  federal  R&D  programs,  or  different  research  project  areas  are  comparatively  rare. 

•  Agencies  differ  in  setting  priorities  for  science  based  on  whether  they  have  a  scientific  or  mission  orientation. 
Most  agencies  now  gather  views  fi'om  various  stakeholders  combined  with  strategic  planning  and  goal-setting. 
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Key  Gaps  in  the  Literature 

While  high-level  goal  setting  is  discussed,  and  the  process  of  peer  review  and  scientific  advice  is  also  detailed,  there 
is  very  little  about  the  vast  middle  ground  where  goals-setting  meets  actual  funding  obligations.  Although  reports  cite  the 
primary  role  of  the  Executive  Office  of  the  President  in  priority  setting  and  coordination,  relatively  little  exists  about  actual 
operations  such  as  the  role  of  the  NSTC  in  coordinating  federal  R&D.  The  literature  cites  NSTC  as  coordinating  larger 
initiatives  and  crosscuts,  although  the  importance  of  it’s  decisions  versus  those  of  0MB  staff  is  not  detaQed.  Likewise,  there 
is  little  description  of  the  NSTC  role,  if  any,  in  determining  funding  in  agency  core  R&D  programs  not  cormected  to  a  larger 
budget  priority  or  “crosscut.” 

There  is  even  less  detail  about  the  process  that  takes  place  within  the  Executive  Branch  in  the  1 1  months  leading  up 
to  the  release  and  explanation  of  the  President’s  budget.  The  deliberation  within  agencies  for  resources  in  the  period  prior  to 
the  submission  of  the  proposed  budget  submission  to  0MB  in  September  is  nowhere  in  the  literature.  Likewise,  the  give- 
and-take  between  0MB  and  the  agencies  prior  to  the  agencies  being  “locked  out”  of  the  budget  in  December  is  not  described. 
There  is  no  mention  of  how  the  individual  divisions  of  0MB  make  decisions,  set  priorities,  or  allocate  funding.  The 
readjustment  of  the  budget  that  occurs  after  the  agency  “lock  out”  in  December,  and  when  final  Presidential  priorities  are  set, 
is  not  described  in  the  literature. 

Furthermore,  there  is  little  description  of  the  ways  in  which  Congressional  committees  influence  the  direction  and 
conduct  of  federal  R&D  though  a  number  of  informal  means.  Rarer  still  are  documents  that  elaborate  on  either  the  details  of 
these  procedures  in  practice  or  the  degree  to  which  the  practice  corresponds  to  formal  procedures. 

Finally,  despite  the  sizable  academic  literature  on  methods  for  assessing  research  benefits,  diere  is  virtually  no 
discussion  of  whether  or  how  these  have  been  implemented  by  the  research-sponsoring  community. 


Best  Practices  in  the  Literature 

The  literature  itself  offers  no  clear  concept  of  best  practice  nor  attempts  formally  to  make  such  an  assessment.  Doing 
so  would  require  establishing  a  metric,  a  task  difficult  to  perform  when  agency  missions  vary  so  greatly.  Yet,  the  literature 
might  be  said  to  imply  a  definition  of  best  practice  by  critiquing  present  practice,  as  discussed  below.  As  noted,  these 
critiques  generally  advocate  some  selective  change  rather  than  offer  an  integrated  design  and  might  be  said  more  to  offer 
views  of  “ideal”  practice  than  identify  best  practice. 

There  are  some  cases  where  the  U.S.  government  has  adopted  some  of  the  recommendations  made  in  different  reports 
but  the  effectiveness  of  these  changes  remains  unclear.  For  example,  the  White  House’s  creation  of  a  “21“  Century  Research 
Fund”  addresses  some  of  the  criticism  that  too  much  development  has  been  lumped  together  with  basic  research.  Responses 
to  the  increased  demand  for  accountability  of  science  and  technology  have  also  affected  priority  setting  practice  in  many 
R&D  agencies.  The  literature  has  yet  to  catch  up  with  these  developments,  but  these  changes  may  be  worth  further 
examination. 
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Government-Wide  Coordination  and  Priority  Setting 

There  is  no  fomally  defined  process  within  the  federal  government  to  set  goals  and  priorities  or  make  allocation 
decisions  for  science.  The  system  is  a  pluralistic  one  based  in  principle  on  promoting  excellence  and  relevance.  Many 
players  with  different  interests  interact  to  influence  the  outcomes.  Recommendations  found  in  the  literature  on  setting  broad 
goals  for  federally  funded  research  fall  into  three  broad  categories: 

•  Tying  science  funding  more  tightly  to  national  goals; 

•  A  science  goals  approach  with  realignment  of  budgets,  if  needed,  to  better  meet  the  needs  of  scientific 
advancement;  and 

•  Fine  tuning  the  existing  complex,  political  process. 

Suggestions  for  more  detailed  models  of  priority  setting  in  turn  may  be  ascribed  to  three  categories: 

•  Pngaging  the  scientific  community  in  determining  priorities  based  on  scientific  needs; 

•  Benchmarking  U.S.  capabilities  and  determining  where  more  emphasis  might  be  placed; 

•  Seeking  scientific  and  stakeholder  input  in  science  to  meet  agency  missions; 


Critiques  of  Currently  Employed  Methodology 

The  most  fi-equent  criticism  addresses  a  perceived  lack  of  clear  methodology  for  performing  priority  setting 
and  coordination.  Enactment  of  GPRA  has  led  to  changes  in  agencies’  practices,  yet  a  further  implicit  critique  may 
be  found  in  the  actions  of  the  House  Science  Committee  which  held  hearings  in  1996  and  1997  on  implementation 
in  the  civilian  science  agencies  and  announced,  in  1997,  that  this  would  be  a  major  oversight  target.  A  major 
argument  in  the  1995  NRC  “Press”  report  is  the  need  for  some  form  of  "comprehensive"  and  "coherent" 
coordination  of  federally-financed  research.  However,  even  this  recommendation  is  by  no  means  universally 

accepted. 


Advantages  and  Disadvantages  of  Different  Methodologies 

Best  practice  in  the  use  of  different  methodologies  suggest  that  a  pluralistic  approach  is  actually  the  more 
rational  way  to  make  determinations  among  competing  priorities.  For  example,  one  argument  against  a  more 
coherent  and  integrated  federal  S&T  budget  suggests  that  trade-offs  should  be  made  at  the  agency  level  between 
S&T  investment  and  other  expenditures;  the  Press  report  underestimates  the  value  of  the  mechanisms  already  in 
place,  especially  the  NSTC;  and  warns  against  the  “overly  comprehensive  process”  proposed  by  the  Press  panel. 

Other  voices  argue  that  the  budget  process  will  not  provide  a  method  or  even  an  analytic  fimnework  for 
setting  the  major  priorities  in  the  budget  because  of  the  diversity  of  agency  goals.  Tbe  current  process  recognizes 
R&D’s  value  and  the  broad  acceptance  of  its  major  federal  role.  Yet,  it  is  too  difficult  to  budget  by  individual 
projects.  The  “level  of  effort”  approach  is  hard  to  defend,  especially  in  light  of  the  difficulty  of  making  causal 


arguments  by  tracking  direct  benefits.  Further,  under  current  practice,  the  fate  of  entire  disciplines  sometimes 
depends  upon  the  funding  decisions  of  individual  agencies. 

Suggested  Approaches  to  Improving  Process 

Alternatives  to  the  present  processes  fall  into  one  of  three  areas:  alternative  weightings  or  other  means  for 
deriving  priorities  from  larger  national  goals;  suggestions  for  alternative  mechanisms  within  existing  institutions; 
and  changes  in  those  institutional  structures  themselves. 

Alternative  Weightings.  In  the  first  area,  there  are  calls  to  clarify  the  raison  d*etre  for  federal  R&D  support. 
There  are  frequent  recommendations,  for  example,  to  link  allocations  more  directly  to  specific  societal  goals. 
Whatever  criteria  are  chosen,  actual  processes  of  selection  and  allocation  should  be  more  explicit.  OTA  provided  an 
example  of  one  set  of  criteria  for  selecting  among  competing  initiatives  summarized  in  Table  S-1. 


Table  S-1.  OTA’s  Suggested  Criteria  for  Selecting  Among  Competing  R&D  Initiatives 


Scientific  Merit 

Scientific  objective  and  significance 

Breadth  of  interest 

Potential  for  new  discoveries  and  imderstanding 

Uniqueness 

Social  Benefits 

Contribution  to  scientific  awareness  or  improvement 
of  the  human  condition 

Contribution  to  international  understanding 

Contribution  to  national  pride  and  prestige 

Programmatic  Concerns 

Feasibility  and  readiness 

Scientific  logistics  and  infinstructure 

Community  commitment  and  readiness 

Institution^  implications 

International  involvement 

Cost  of  proposed  initiative 

In  addition  to  priorities  set  by  issue  area,  there  are  also  calls  to  do  so  by  stage  of  the  research  and  innovation 
process  or  other  criteria.  Similarly,  there  are  also  suggestions  to  shift  the  focus  of  funding  in  the  federal  R&D 
portfolio  dramatically  toward  basic  research  while  others  warn  that  parsing  the  federal  R&D  budget  by  the  old 
definitions  of  basic  and  applied  research  has  proved  politically  ineffective. 

Alternative  Mechanisms.  The  second  major  group  of  alternatives  addresses  the  mechanisms  by  which 
allocations  should  occur  within  existing  institutions.  The  concept  of  best  practice  might  be  applied  by  adapting 
already-existing  models  to  other  federal  agencies.  Several  suggestions  have  been  made  for  fundamental  changes  in 
allocation  methods.  One  would  use  an  options  approach  where  the  portfolio  is  constantly  updated,  balanced  for  risk, 
and  takes  advantage  of  increased  information  availability.  Many  view  the  current  system  as  largely  successful  for 
the  bulk  of  research  needs  but  suggest  that  within  a  pluralistic,  multi-agency  budgeting  approach,  some  areas  require 
special  attention  owing  to  their  large  potential  for  spillover  effects  to  other  agencies.  Several  reports  point  to  the 
paucity  of  data  gathering  and  the  necessity  for  establishing  a  database  of  the  federal  R&D  budget. 

Structural  Changes.  The  last  category  of  suggestions  addresses  the  institutions  of  federal  research  portfolio 
management  themselves.  Several  studies  advocate  a  greater  role  for  NSTC,  OSTP,  and/or  OMB  in  setting  portfolio 
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guidelines.  This  would  constitute  a  fimdamental  redrafting  of  the  role  of  ftiese  agencies  and  the  nature  of  their 
interactions  with  the  rest  of  the  federal  research  portfolio  management  structure.  Improved  coordination  could 

require 

•  a  comprehensive,  comparable  data  base  on  R&D  budgets; 

•  a  detailed  “directory”  report  to  provide  information  on  what  agencies  are  engaged  in  what  kinds  of 
R&D;  and 

•  a  report  on  “R&D  in  the  Budget”  each  year. 

Alongside  suggestions  for  different  goals  stands  the  suggestion  that  a  new  institutional  structure  be  created, 
such  as  a  non-governmental  National  Fonim  on  S&T.  Such  a  body  might  also  define  what  the  essential  elements  in 
the  federal  R&D  portfolio  must  be  and  suggest  ways  in  which  the  portfolio’s  composition  may  be  more  readily 
adapted.  Some  proposals  call  for  creating  a  Federal  S&T  Budget  in  lieu  of  the  existing  post  hoc  accounting  concept 
and  also  shifting  fi-om  a  bottom-up  to  a  top-down  process.  This  would  force  trade-offs  at  the  programmatic  level. 

Yet,  at  the  same  time  there  are  voices  stating  that  the  cunent  process  of  trade-offs  and  political  decisionmaking, 
influenced  by  advocates  of  science,  actually  works  fairly  well  and  meets  the  needs  of  science  for  adequate  funding. 

DEFINING  AND  DETAILING  "R&D"  AND  “S&T” 

Defining  R&D 

From  the  outset,  a  terminology  problem  confounds  attempts  to  characterize  the  literature  on  prionty  settmg  and  R&D. 
Although  the  terms  S&T  and  R&D  are  often  used  interchangeably,  they  have  very  different  meanings  in  the  context  of  the 
federal  government.  Specifically,  there  is  one  overarching  and  official  definition  of  R&D  used  by  all  federal  agencies* . 
Because  R&D  activities  constitute  the  primary  long-term  investment  of  the  federal  government  (education  and  training  is 
number  two),  R&D  is  separately  tracked  in  the  federal  budget.  2  Complicating  the  discussion  is  the  feet  that  other  terms 
have  been  introduced,  including  the  “Federal  Science  and  Technology  Budget”  and  the  “21“  Century  Research  Fund.” 

Figure  S-1  shows  the  relationship  between  these  three  terms. 

1  R&D  is  a  budget  category  that  is  defined  by  the  Office  of  Management  and  Budget  (0MB)  in  Section  25  of  Circular  A-1 1 . 

2  The  Department  of  Defense  (DOD)  alone  among  the  federal  agencies  has  refined  the  0MB  definition  by  sub-dividing  the 
Development  category.  DOD  then  takes  one  of  these  sub-categories,  groups  it  with  Basic  Research  and  Applied  Research,  and  collectively 
refers  to  these  three  activities  as  “S&T”  thus  designating  S&T  as  a  sub-set  of  R&D. 
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Figure  S-1:  The  National  Academy  of  Sciences  Comparison  of  3  “R&D”  Budgets 


FIGURE  2:  FY  2001  R&D,  FS&T,  and  21“  Century  Rescan*  Fund  (in  bimons) 


R&D=$^5.4biffi(m 


FS&T  =  $53.7billion 


21**  Cetrtuiy  Research  Fund  =  $42.9  biUion 


Source;  Research  &  Development  FY260OI  .  Washington,  DC:  American  Association  for  the  Advancement  of  Science,  April  2000. 


Defining  S&T 

Parallel  to  the  designation  of  specific  federal  activities  as  R&D,  there  is  a  simultaneous  labeling  as  to  general 
purpose  (l.e.,  mission)  or  function  activities  for  S&T.  R&D  activities  are  found  in  eveiy  functional  categoiy  in  the 
federal  budget.  Of  particular  interest  are  the  R&D  activities  in  Function  250  -  General  Science,  Space  and 
Technology.  All  such  activities,  most  especially  those  of  NSF  and  much  of  DOE,  are  officially  labeled  as  S&T 
activities.  Only  a  part  of  the  activities  are  categorized  as  R&D.  Consequently,  for  these  agencies,  R&D  is  a  sub-set 
of  S&T. 

The  magnitude  of  civilian  agency  S&T  activities  is  hard  to  determine,  because  they  are  not  officially 
labeled  S&T.  Figure  2  illustrates  that  specific  activities  that  are  widely  believed  to  be  R&D  are  instead  S&T 
activities  that  fall  outside  the  set  of  activities  officially  designated  as  R&D  (e.g.,  the  Manufacturing  Extension 
Program  at  NIST  and  the  Space  Shuttle).  Failure  to  agree  on  the  definition  of  critical  terms  and  then  apply  them 
consistently  has  defeated  and  continues  to  defeat  basic  communication  in  the  federal  R&D  community. 

Data  Issues 

The  way  R&D  is  defined  affects  the  collection  and  sharing  of  government  data.  Data  on  the  contents  of  the 
federal  R&D  portfolio  contain  either  highly  aggregated  budget  information  or  disaggregated  project  descriptions. 
There  is  considerable  difficulty  finding  common  bases  for  combining  “crosscutting”  data  collected  by  different 
agencies.  Moreover,  activities  not  characterized  as  R&D  but  which  are  scientific  in  nature  (i.e.,  weather  data,  space 
travel,  mapping)  are  not  included  in  descriptions  of  federal  R&D  activities,  leading  to  some  confusion  during 
priority  setting  and  coordination  activities. 
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Figure  S-2  Contrasting  Defense-Related  and  Civilian  Definitions  of  S&T 


EXECUTIVE  BRANCH  ORGANIZATIONS 

The  actual  operation  and  effectiveness  of  executive  branch  organizations  and  processes  for  coordinating 
R&D  policy,  planning,  and  funding  are  poorly  described  in  the  literature.  Most  of  the  material  included  here  is 
derived  from  agency  procedural  documents. 

The  Executive  Office  of  the  President 

The  Executive  Office  of  die  President  has  four  offices  or  councils  that  advise  the  President  about  priority 
setting  in  R&D  and  S&T.  These  are: 

•  The  Office  of  Science  and  Technolt^y  Policy.  OSTP  helps  coordinate  federal  science  activities  to  meet 
the  President’s  goals.  This  is  primarily  done,  in  the  Clinton  Administration,  through  the  National  Science 
and  Technology  Council  (NSTC)  for  which  OSTP  acts  as  a  secretariat.  OSTP,  together  with  the  Office  of 
Management  and  Budget,  issues  a  budget  memorandum  each  year  on  research  and  development  priorities. 

•  President’s  Council  of  Advisors  on  Science  and  Technology  (PCAST).  PCAST’s  principal  task  is  to 
assist  the  NSTC  in  securing  private  sector  involvement  in  the  latter’s  activities.  Some  of  its 
recommendations  are  general,  but  PCAST  also  makes  specific  recommendations  based  upon  assembled 
panels  as  well  as  its  own  reviews  of  reports  of  the  NSTC. 
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•  National  Science  and  Technology  Council  (NSTC)  was  established  to  integrate  the  President’s  S&T 
policy  agenda  across  the  federal  government  and  ensure  that  S&T  is  considered  in  development  and 
implementation  of  federal  policies  and  programs.  It  is  a  policy  and  budgetary  coordination  body  through 
which  all  executive  departments  and  agencies  coordinate  S&T  activities  that  require  significant  levels  of 
interagency  coordination.  OSTP  suggests  topics  aroimd  which  the  NSTC  forms  committees  to  review 
government  spending  in  specific  areas  of  research  and  recommend  priority  or  allocation  shifts.  OSTP  then 
advises  agencies  and  OMB  and  solicits  input  from  the  larger  scientific  community  about  where  priorities 
and  resource  allocation  should  focus.  In  preparation  for  FY2001,  NSTC  is  overseeing  the  coordination  and 
priority-setting  for  1 1  areas  of  which  the  more  mature,  congressionally-mandated  programs  are  managed  as 
formal  interagency  crosscuts,  while  areas  being  developed  for  priority  attention  become  the  subject  of 
NSTC  worldng  groups. 

•  The  Office  of  Management  and  Budget  (OMB)  coordinates  the  President’s  budget  process.  This  process 
starts  each  summer  when  agencies  begin  preparing  tiieir  budgets  for  the  fiscal  year  that  begins  in  October 
of  the  following  year.  There  is  no  “R&D  budget”  as  such.  OMB  has  the  power  to  shape  the  budget,  but 
does  not  set  specific  priorities  for  science.  It  does  examine  agency  proposals  for  redundancy  and  looks  for 
opportunities  for  interagency  coordination.  It  has  no  means  for  truly  setting  priorities  between  different 
R&D  programs  with  differing  goals.  Budget  guidelines  for  R&D  are  issued  jointly  by  OSTP  and  OMB. 
While  agency  officials  report  that  the  budget  call  does  have  some  effect  on  R&D  allocation,  it  may  actually 
affect  more  how  existing  plans  are  labeled  than  on  how  budget  priorities  are  set. 

The  priorities  for  R&D  that  become  guidelines  for  the  agency  budgeting  process  are  based  on  a  set  of  goals 
named  by  the  Clinton  Administration  in  the  first  months  of  its  tenure.  These  goals  include:  (1)  a  healthy,  educated 
citizenry,  (2)  job  creation  and  economic  growth,  (3)  world  leadership  in  science,  mathematics,  and  engineering,  (4) 
improved  environmental  quality;  (5)  harnessing  information  technology;  and  (6)  enhanced  national  security.  The 
R&D  priorities  have  remained  relatively  stable  over  the  past  six  years,  with  several  additions,  as  illustrated  in  Figure 


S-3  below. 
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Figure  S-3.  R&D  Priorities  Set  by  the  OSTP  and  OMB,  ry96-FY01 

FY96 _ m? _ FY98 _ R[99 _ FYOO _ FY01 

Partnership  for  a  new  generation  of  vehicle _ _ 

Biomedical  research,  health  promotion  and  disease  and  injury  prevention  research _ 

Learning  and  cognitive  processes _ 

Materials  technology _ 

Energy  production  and  utilization  technologies _ 

Integrated  ecosystem  management _ 

Networking  and  communications _ 

Human-computer  interaction _ 

Counter-terrorism _ _ 

Detection,  monitoring,  and  verification _ 

Civilian  aircraft _ _ 

Telemedicine _ _ 

Infectious  diseases _ _ 

Environmental  risk  assessment _ 

Microelectronics _ 

I  U.S.  Global  Climate  Change  Research 

I  Plant  genome  research 

I  Nanotechnology 


R&D  Agencies  Withn  the  Executive  Branch 

The  literature  on  agency-level  R&D  priority  setting  is  not  a  robust  collection.  The  Office  of  Technology 
Assessment  report  “Federally  Funded  Research:  Decisions  for  a  Decade”  describes  these  activities,  and  diis  section 
draws  heavily  from  that  report.  Beyond  this,  the  agencies  themselves  have  issued  GPRA-inspired  strategic  plans 
that  provide  some  insights  into  the  priority  setting  process.  Outside  of  these  sources,  we  found  very  little  concerning 
what  happens  in  the  agencies  with  regard  to  priority  setting,  despite  there  being  over  20  government  agencies 
funding  R&D.  It  makes  sense  that  the  largest  spenders  would  be  the  most  well  represented  in  the  literature,  but 
smaller  agencies  most  likely  make  dearer  trade-offs  in  funding.  These  smaller  agencies  may  be  worth  further 
examination.  Table  S-2  below  siunmarizes  what  exists  in  the  literature  about  agency  priority  setting  activities.  Not 
all  of  these  representations  may  be  current.  Some  of  the  literature  is  dated  and  changes  may  well  have  occurred  in 
these  agencies  since  the  original  report  was  written. 


Table  S-2.  Agency  R&D  Priority  Setting  Activities 


AGENCY  (IN  ORDER 

OF  THE  MAGNITUDE 
OF  THEIR  R&D 
BUDGETS) 

PRINCIPAL  FINDINGS  ABOUT  PRIORITY  SETTING 

AcnvrriES  reported  in  literature 

METHODS  USED  TO  IDENTIFY 

PRIORITIES 

Department  of  Defense 

Planning  occurs  in  the  office  of  the  Director  of  Defense 

Research  and  Engineering  (DDR&E)  which  looks  to  the 

NSTC  and  the  Joint  Chiefs  of  Staff’s  Joint  Vision  2010  for 
guidance. 

In  its  Basic  Research  Plan,  DOD  uses  peer  review 
and  competition  to  achieve  its  objectives; 

Technology  Area  Reviews  and  Assessments 
(TARA)  provide  an  oversight  fimetion  to  assess 
the  quality  of  the  research  programs. 

Department  of  Energy 

A  National  Energy  Strategy  (NES)  was  designed  to  solicit 
input  from  the  offices  within  DOE  and  from  external  advisors. 
Each  program  has  an  advisory  panel,  such  as  the  High  Energy 
Physics  Advisory  Panel  (HEPAP)  and  the  Energy  Research 
Advisory  Board  (ERAB)  which  are  external  boards  of 
scientists.  These  groups  and  others  like  them  present  to  DOE 
a  set  of  priority  research  areas  that  deserve  the  ^ency’s 
special  attention. 

In  selecting  areas  of  research,  the  Office  of 

Science  emphasized  the  use  of  peer  review  to 
evaluate  all  programs.  It  also  stated  that  advisory 
boards  play  a  significant  part  in  its  priority  setting 
processes. 

NASA 

NASA  sets  priorities  in  conjunction  with  the  budget  process 
and  by  selecting  specific  projects.  Influenced  more  heavily  by 
Congress  than  other  agencies.  The  process  is  essentially 
bottom  up  with  project  managers  proposing  new  initiatives. 
When  large  missions  are  proposed,  such  as  Space  Station 
Freedom,  top-down  direction  determines  the  parameters  of  the 
effort  (OTA) 

Priority  setting  results  from  a  combination  of 
input  from  NASA’s  own  internal  managers,  staff 
and  directors,  and  external  actors  like  the  National 
Research  Council  the  Task  Group  on  Space 
Astronomy  and  Astrophysics,  and  the  space 
science  community.  In  its  goal  to  pursue  scientific 
excellence,  the  Office  of  Science  emphasized  the 
use  of  peer  review  to  evaluate  all  programs. 

National  Institutes  of 

Health  (HHS) 

The  director  of  each  Institute,  with  the  help  of  NIH’s  national 
advisory  council,  decides  funding  direction  carried  out 
through  extramural  grants  and  intramural  programs. 

Advisory  councils  are  mandated  by  Congress  and 
composed  of  people  from  bodi  the  scientific 
community  and  the  public.  The  director  also 
consults  with  intramural  investigators  ,scientists  in 
the  extramural  program,  patients  and  their  families 
interested  in  research  on  particular  diseases, 
professional  and  scientific  groups,  representatives 
of  the  Administration  and  members  of  Congress, 
and  with  the  public. 

National  Science 
Foundation 

The  NSF  process  for  strategic  planning  involves  calling  in 
advisory  committees  and  committees  of  visitors,  regular 
reviews  of  programs  and  input  from  the  National  Science 

Board,  and  at  the  Directorate  level  reports  from  external 
groups  on  program  issues.  Goals  are  set  “by  scientific 
opportunity  and  the  proposal  process,  as  well  as  in  special 
initiatives  from  advisory  panels.” 

The  NSB  recommended  that  the  following  two 
criteria  be  adopted  in  place  of  the  four  criteria  that 
had  been  used  in  the  past  to  determine  research 
priorities: 

•  What  is  the  intellectual  merit  of  the  proposed 
activity?  E.g.,  does  it  advance  knowledge  and 
understanding  in  its  own  field  and  across  fields?  Is 
it  creative  and  original? 

•  What  are  the  broader  impacts  of  the 
proposed  activity?  E.g.,  advance  discovery  and 
promote  teaching?  Enhancing  partnerships? 

Department  of 

Agriculture 

USDA  derives  specific  priorities  from  its  1997  strategic  plan. 
Annual  performance  plans  are  modified  based  on  input  from 
the  staff  and  advisory  committees.  Priority  setting  is  advised 
by  many  groups,  most  important  is  the  Joint  Council  on  Food 
and  Agriculture  Sciences  created  by  Congress. 

Budgets  are  developed  using  a  crosswalk  that 
links  the  strategic  goals  and  objectives  of  the 
agency  with  its  overall  budget  structure  and 
specific  performance  goals. 

Department  of 

Commerce,  National 
Institute  for  Standards  and 
Technology 

NIST  sets  priorities  in  specific  measurement  areas  based  on 
the  advice  of  councils  created  by  NIST  itself  but  which  are 
established  as  independent  nonprofit  organizations  as  well  as 
input  from  customers  and  NIST  scientific  and  technical  staff. 

The  councils  strive  to  provide  a  consensus  on 
industrial  and  academic  requirements  for 
standards  and  programs,  including  setting 
priorities.  Divisions  maintain  direct  contact  with 
customers  and  manufacturers  and  conduct 
periodic  customer  surveys  in  order  to  set  priorities 
based  on  customer  need. 

Environmental  Protection 
Agency 

EPA’s  Office  of  Research  and  Development  (ORD)  has  the 
principal  responsibility  for  research  and  development. 

Strategic  Plans  have  relied  heavily  on  EPA’s  Science 

Advisory  Board  (an  independent  group  of  engineering  and 
science  advisors)  and  expert  panels  convened  by  NAPA  and 
the  National  Academy  of  Sciences 

The  most  important  of  EPA’s  strategic  principles 
is  the  explicit  use  of  the  risk  paradigm  to  shape 
and  focus  EPA’s  organizational  structure  and 
research  agenda,  including  hazard  identification, 
dose  response  assessment,  exposure  assessment, 
and  risk  characterization. 
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CONGRESS 

Of  all  the  institutions  involved  in  coordination  and  priority  setting  across  fields  of  science,  the  literature  as 
a  whole  shows  its  largest  gap  in  its  treatment  of  Congress.  Congress  has  not  paralleled  the  Executive  branch  m 
coordinating  its  own  R&D  policy,  planning,  and  funding  efforts.  At  least  21  Congressional  Committees  have  direct 
federal  R&D  policy  or  funding  responsibility.  At  no  time  in  the  Congressional  process  is  there  a  comprehensive 
view  taken  of  the  R&D  portfolio  across  the  federal  government.  Further,  there  are  a  range  of  other  legislative 
decisions  that  can  affect  plaiming  and  priorities  of  federal  agencies  and  the  conduct  of  federally-funded  R&D. 
Regulatory,  tax,  or  other  decisions  affecting  research  institutions  are  made  outside  of  the  Congressional  circles  in 
which  R&D  policy  is  decided  and  are  ficquently  not  coordinated  with  the  Congressional  entities  having  R&D  policy 

jurisdiction. 

INTERACTIONS  BETWEEN  THE  EXECUTIVE  AND  LEGISLATIVE  BRANCHES  ON  R&D 

The  need  for  interaction  between  branches  causes  the  shortcomings  found  within  each  to  become 
compounded  by  the  lack  of  a  formal  coordinating  mechanism.  Since  0MB  must  approve  agency  testimony,  any 
formal  presentation  to  Congress  will  serve  to  ratify  the  final  decisions  made  during  the  Budget  submission  and  will 
not  easily  provide  a  vehicle  for  an  R&D  agency  to  comment  critically  upon  decisions  made  in  that  process.  In 
addition,  agencies  are  generally  nonresponsive  to  questions  of  priority  put  to  them  by  Congress.  Yet,  oversight 
hearingJplay  a  key  role  in  determining  the  budgets  of  specific  research  programs  and  encouraging  coordination 
between  the  research  agencies.  Absent  a  formal  agreement,  the  budget,  appropriations,  and  oversight  processes 
constitute  the  coordination  mechanisms  and  the  conduct  of  this  work  is  subject  to  individual  agency  and  committee 
(fynamics  and  is  frequently  left  to  the  perspectives  and  proclivities  of  individual  members  of  Congress  and  then- 

staffs. 

USE  OF  BENEFITS  MEASURES  IN  PRIORITY  SETTING  AND  BUDGET  COORDINATION 

With  the  introduction  of  new  concepts  of  accountability,  the  R&D  agencies  have  begun  applying  benefits 
measures  to  R&D  and  using  the  results  to  help  set  priorities.  However,  this  process  has  not  been  studied  or 
systematically  documented.  It  may  be  too  early  in  the  process  of  adopting  these  measures  to  determine  if  they  are 
effective.  Most  of  die  measures  identified  in  the  literature  were  adopted  from  private  sector  applications.  In  many 
ways,  federal  research  presents  greater  problems  for  measurement  and  benchmarking  than  does  pnvate  R&D.  A 
great  deal  of  federally  funded  research  is  directed  to  areas  where  the  market  is  limited  at  best.  Further,  given  the 
types  of  data  available,  the  returns  that  result  from  most  calculations  must  be  interpreted  as  average  rather  than 
marginal  rates.  From  a  policy  perspective,  this  means  we  cannot  be  certain  from  this  aggregate  analysis  what  the 
effect  of  an  additional  dollar  of  research  expenditure  might  be.  The  cost/benefit  framework  itself  may  be  too 
restrictive,  failing  to  capture  the  many  benefits  that  may  be  derived  from  publicly-funded  basic  research.  The  true 
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effect  of  such  outlays  may  well  be  indirect,  affecting  productivity  through  changing  the  returns  to  private  research 
and  development  rather  than  directly  as  a  result  of  the  specific  research  project. 

Social  rates  of  return  analysis  seeks  to  determine  the  sum  of  benefits  accrued  from  changes  in  the  knowledge 
base  and  compares  these  benefits  to  the  cost  of  investment.  This  social  benefit  may  be  considerably  greater  than  the 
private  benefit  taken  in  the  form  of  profit.  As  a  practical  matter,  such  studies  involve  selecting  a  sample  of  specific 
iimovations  upon  which  to  perform  these  calculations.  This  is  both  expensive  and  subject  to  unintentional  or 
unavoidable  bias  in  the  selection  process.  Further,  the  social  rate  of  return  calculated  by  such  means  is  not  directly 
comparable  to  the  internal  rates  of  return  calculated  for  private  investment  projects.  Nevertheless,  studies  in  this 
area  have  found  a  very  high  return  to  investment  in  basic  research. 

Among  potential  users  of  such  information  is,  of  course.  Congress.  Whether  it  acts  as  the  originator  of 
information  requests  for  the  purposes  of  furthering  its  own  process,  or  is  targeted  as  the  ultimate  audience  for 
assessments  produced  for  its  benefit  by  the  agencies  coming  before  its  committees  for  funding.  Congress  would  like 
to  have  better  means  for  determining  the  results  ensuing  from  federal  funding  of  R&D.  Another  body  which  has 
considered  broad  application  of  performance-based  measures  throughout  the  federal  government  is  the  NSTC. 

NSTC  has  issued  a  list  of  performance  measures  for  fimction  250  (R&D)  activities  that  encourage  setting  aside  80 
percent  or  more  of  R&D  for  peer-reviewed  competition  as  well  as  call  for  the  majority  of  assessments  to  be  made  by 
external  bodies.  These  are  in  reality  guidelines  for  conduct  and  measures  attuned  to  the  first  category  found  in  the 
literature:  asset-based  measures. 

The  National  Academy  of  Sciences  Committee  on  Science  Engineering  and  Public  Policy  (COSEPUP,) 
suggests  general  guidelines  for  measuring  U.S.  position  in  a  given  field  of  science:  1)  What  is  the  U.S.  position  in  a 
field?  2)  What  key  fectors  determine  relative  position?  and  3)  What  is  the  trend  for  relative  position  in  the  near  and 
long  term?  These  yardsticks  do  not  necessarily  track  well  with  the  needs  of  mission  agencies.  Further,  in  practice, 
most  measures  appear  to  be  of  the  asset-accounting  type. 

At  the  agency  level,  the  DOE  Office  of  Science  Strategic  Plan  lists  a  series  of  success  indicators  for  each  of 
its  five  main  goal  areas  (e.g.:  “photochemical  systems  that  hold  promise  for  economical,  highly  efficient  solar 
cells.”)  The  indicators  are  outcome-oriented  but  seem  to  be  of  a  checklist-type,  attuned  to  achieving  particular 
milestones  and  not  quantitative  in  nature.  They  do  not  seek  to  track  direct  benefit  back  to  specific  R&D  project 
outcomes 

Documents  on  the  performance  assessment  process  in  use  by  NSF  point  to  heavy  reliance  on  external 
assessment.  Attempts  are  being  made  to  shift  from  a  somewhat  ad  hoc  basis  to  a  more  formal  procedure  tiiat  will 
provide  a  common  format  to  the  review  process  that  extends  across  NSF.  For  example,  beginning  in  1998,  annual 
reports  to  the  Director  have  been  required  from  NSF  units  using  a  variety  of  indicators  and  data  series  to 
demonstrate  effect. 

Generally  speaking,  however,  although  new  advances  continue  to  be  suggested  in  the  academic  literature  and 
new  methodologies  for  identifying  and  selecting  new  research,  managing  existing  research,  and  evaluating  and 
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3«;pi«!<!iTig  research  retrospectively  have  been  designed,  the  implementation  of  these  mediods  by  the  research 
sponsoring  community  remains  minimal. 

CONCLUSIONS 

The  alternative  approaches  for  managing  the  federal  research  enterprise  that  emerge  from  the  literature  fall 
into  three  areas: 

•  alternative  readings  of  what  are  the  appropriate  goals  federal  research  support  should  seek  to  fulfill; 

•  suggestions  for  alternative  mechanisms  for  allocation  of  funds  within  the  existing  institutions  for 
managing  the  federal  research  portfolio;  and 

•  elements  for  a  design  leading  in  whole  or  in  part  to  changes  in  those  institutional  structures  themselves. 
The  two  cons’gtfint  themes  are  a  desire  to  establish  priorities  and  to  do  so  in  a  coordinated  fashion.  This  assumes 
fliat  improvements  in  either  the  top-down  or  the  bottom-up  approach  or  both  would  improve  the  outcome.  This 
assumption  has  not  been  questioned  in  the  literature. 

This  is  understandable.  A  frequent  theme  of  the  literature  is  that  the  federal  R&D  portfolio  is  only  &post 
facto  accounting  concept.  It  is,  by  default,  the  aggregation  of  individual  mission  agency  portfolios  but  is  in  no  sense 
managed  ex  ante  as  a  unified  portfolio.  Several  studies  advocate  a  greater  role  for  the  NSTC,  OSTP,  and/or  OMB  in 
setting  portfolio  guidelines  at  a  higher  level  than  that  of  the  funding  agencies  as  well  as  actively  monitoring 
fulfillment.  A  necessaiy  first  step  to  effective  prioritization,  in  other  words,  is  to  achieve  coordination. 

Nevertheless,  the  issue  of  whether  a  more  unified  or  better  coordinated  portfolio  is  desireable  or  achieveable  has  not 
been  adequately  debated  in  the  literature  and  deserves  more  attention. 

A  persistent  assumption  in  the  literature  is  that  greater  coordination  is  desirable  and  can  be  attained  by 
setting  high  level  goals  and  then  proceeding  to  lower  levels  of  decisionmaking  authority.  This  seems  problematic  on 
two  counts.  First,  decisionmaking  in  this  area  is  embedded  in  existing  institutions  and  political  processes.  Setting 
high-level  goals  and  then  rigorously  enforcing  them  as  the  means  for  crafting  priorities  and  making  allocations  on 
lower  levels  would,  in  effect,  stand  the  present  system  on  its  head. 

Second,  such  an  approach  may  not  accord  with  the  evolving  pattern  for  the  commerce  of  ideas  and 
knowledge  management.  Modem  science  is  increasingly  cross-disciplinary,  with  major  discovery  taking  place  at 
the  interstices  of  traditional  disciplinary  categories.  Contributing  to  this  trend  are  increased  globalization  of  effort 
with  geographically  dispersed  working  teams  crossing  geographic  boundaries  and  an  ever  denser  connectivity  of 
information  and  ideas. 

This  is  not  to  suggest  that  prioritization  or  coordination  are  undesirable  or  that  gaining  a  measure  of  control 
is  impossible.  Rather,  it  is  to  suggest  that  as  we  proceed  further  along  this  line  of  inquiry  we  should  address  the 
following  questions  left  largely  unadressed  by  the  literature  as  it  stands: 

1 .  What  do  we  mean  by  the  terms  “priority”  and  “coordination”? 

2.  What  do  we  hope  to  achieve  and  how  will  we  improve  the  public’s  lot  through  prioritization  and 
coordination? 

3.  What  are  appropriate  measures  for  identifying  best  practice  in  priority  setting  and  coordination? 
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4.  What  alternative  models,  not  necessarily  predicated  upon  traditional  views  of  either  the  science  process 
or  its  effect  on  the  larger  society,  need  we  consider  to  best  develop  means  for  achieving  a  true  ability  to 
set  priority  and  the  level  of  coordination  we  desire? 

In  order  to  fully  understand  the  processes  that  take  place  within  the  system  that  result  in  the  set  of  activities 
that  the  government  labels  “R&D”  or  “S&T”  the  Board  needs  a  better  understanding  of  what  is  happening  in  the 
agencies  or  in  the  scientific  community  in  that  “vast  middle”  between  high  level  goals  and  bottom-up  input.  The 
decisions  made  at  the  program  and  project  level  have  not  been  studied  or  described  in  die  literature.  Insights  that 
could  be  gained  from  an  examination  of  these  activities  may  aid  the  Board  in  its  effort  to  bring  more  accountability 
and  coordiantion  to  the  process. 
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1.  INTRODUCTION  AND  METHODOLOGY 


PURPOSE  OF  TfflS  REVIEW 

The  National  Science  Board  is  presently  exploring  how  the  U.S.  federal  government  sets  priorities  in 
research  and  development  and  whether  changes  are  needed  in  the  decisionmaking  process.  Accordingly,  the 
National  Science  Board  Committee  on  Strategic  Science  and  Engineering  Policy  Issues  has  identified  the  need  for  a 
comprehensive  review  of  the  relevant  literature  and  experience  on  R&D  priority  setting  across  fields  of  science\ 
The  purpose  of  this  study  is  to  provide  input  to  the  committee’s  consideration  of  the  options  for  priority  setting 
methodologies  across  fields  of  science  to  improve  the  federal  R&D  budget  allocation  process.  This  task  is  directed 
toward  identifying  and  examining  descriptions  of  the  range  of  budget  coordination  and  priority  setting 
methodologies  that  are  currently  employed  by  the  federal  government  as  well  as  normative  critiques  of  currently 
employed  methodologies. 

RAND  was  asked  to  examine  the  literature  for  insights  into  a  set  of  questions  posed  by  the  NSB.  These 
questions  were  posed  in  die  original  statement  of  work  and  elaborated  upon  in  subsequent  interaction  between  die 
project  team  and  NSB  staff.  Taking  diese  inputs  together  and  placing  them  in  a  imiform  fimnework  has  yielded  a 
hierarchical  oudine  of  broad  issues.  In  brief,  the  literature  in  this  field  was  searched  to  provide  insight  into  the 
following  points: 

•  How  does  the  literature  describe  the  elements  and  structure  of  the  system? 

•  What  does  the  literature  say  about  the  functioning  (measures/methods/models)  of  the  system? 

•  Models  of  approaches  to  priority  setting  presented  in  the  literature 

•  What  critiques  and  suggestions  exist  in  the  literature? 

As  will  be  seen,  not  all  of  the  enumerated  points  were  equally  well  illuminated  by  the  existing  literature  on 
die  subject.  In  some  cases,  the  literature  as  it  stands  was  notably  silent.  Therefore  different  sections  will  be  each 
handled  in  a  manner  appropriate  to  the  level  of  discussion  found  in  the  literature.  In  each  instance  the  discussion 
will  provide  a  substantive  overview  of  the  search  results  as  well  as  an  assessment  of  the  depth  to  which  particular 

issues  are  covered. 


3  As  will  be  discussed  in  the  Special  Issues  Section  1,  there  is  a  tendency  for  the  terms  “R&D”  and  “S&T”  to  be  used  as 
synonyms  in  the  literature  -  even  in  government  documents.  This  report  addresses  itself  to  the  coordination  and  prioritization  of 
Federal  support  to  research  in  science.  Yet,  we  have  mostly  followed  the  usage  found  in  the  literature  we  report  By  preference 
we  have  used  the  term  “R&D”  when  referring  to  the  formal  apparatus  or  process  by  which  such  activities  are  funded. 
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OUTLINE  OF  THE  REVIEW 

What  follows  is  a  systematic  review  of  recent  literature  on  themes  identified  by  the  Committee  as  being  of 
interest  for  its  own  deliberations  and  for  those  of  the  National  Science  Board  as  a  whole.  After  this  introduction  and 
methodology  section,  the  report  presents  an  overview  of  literature  describing  the  roles  of  different  parts  of  the 
federal  government  as  they  interact  to  influence  the  direction  of  R&D. 

Section  two  provides  a  narrative  summaiy  on  the  major  institutions  and  organizations  -  apart  fi-om  the 
agencies  charged  with  managing  portions  of  the  federal  R&D  portfolio  -  who  play  a  role  in  coordination  and 
priority  setting  on  die  large  scale.  Section  three  moves  the  literature  review  to  the  level  of  the  agencies  and  presents 
a  brief  review  of  what  the  literature  presents  as  meftiods  by  which  priorities  are  determined  in  each  as  well  as  to 
review  cases  of  specific  cross-agency  research  initiatives.  Section  four  presents  critiques  of  the  current  process  and 
suggestions  for  change  as  they  as  discussed  in  the  literature.  Section  five  departs  from  a  strict  reading  of  the 
literature  to  make  observations  and  suggestions,  including  those  fi-om  the  authors. 

Special  Issues  Sections 

Two  “special  issues”  sections  follow  this  discussion:  the  first  calls  the  reader’s  attention  to  the  presence  of  a 
serious  definitional  issue  underpiiming  the  NSB’s  larger  inquiry.  In  order  to  address  the  questions  raised  by  the 
Committee  and  the  Board,  an  understanding  of  the  usage  of  the  terms  “science  and  technology”  (“S&T”)  and 
“research  and  development”  (“R&D”)  is  crucial.  Failxu-e  to  recognize  both  tire  distinctions  between  the  two  terms 
and  the  lack  of  cross-agency  consistency  in  ftieir  use  could  prove  sufficient  to  fhistrate  the  best-intended  effort  to 
improve  mechanisms  for  priority-setting  and  budget  coordination  across  agencies  and  across  fields  of  science.  This 
section  also  addresses  the  question  of  relevant  data  sources.  The  second  special  issues  section  considers  in  brief  the 
alternative  measures  proposed  for  assessing  benefits,  the  extent  to  which  the  literature  suggests  they  are  employed, 
and  identifies  the  databases  that  exist  in  various  locales  within  the  federal  government  which  might  assist  in  more 
coordination,  systematic  prioriy  setting,  and  benefits  assessment. 

Attached  to  this  report  is  an  annotated  bibliography  of  sources  relevant  to  the  NSB  deliberations.  While  the 
report  itself  presents  a  thematic  treatment  of  the  literature  in  overview,  the  annotated  bibliography  provides  a  means 
to  consider  the  contributions  of  individual  papers  and  reports. 

Methodology 

This  section  describes  the  methods  used  to  define,  collect,  and  review  the  literature  covered  in  this  study.  In 
order  to  identify  the  universe  of  literature  relevant  to  this  review,  a  set  of  initial  core  sources  was  obtained.  The 
National  Science  Board  in  the  original  task  statement  nominated  these  core  sources.  We  examined  the 
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bibUographies  of  these  core  sources  and  added  relevant  materials  to  the  list.  In  addition,  we  searched  the  RAND  and 
the  Science  &  Technology  Policy  Institute  libraries  and  the  personal  holdings  of  the  analysts  associated  with  this 
effort  for  additional  materials.  Several  earlier  projects,  including  a  National  Academy  of  Sciences  study ,  had 
developed  bibliographies  that  could  be  used  for  this  project,  and  we  relied  heavily  upon  these. 

From  the  initial  list,  we  then  developed  a  set  of  terms  to  be  keyed  into  various  search  engines.  With  this  new 
material,  we  iterated  our  search  methodology  by  examining  the  bibliographies  of  these  reports  and  articles  for  any 
other  relevant  materials.  At  the  same  time,  we  reviewed  the  AAAS  Science  and  Technology  Policy  Yearbooks  for 
articles  that  would  have  relevance  to  the  questions  being  asked  by  the  NSB. 

Once  the  results  were  processed,  the  project  team  examined  the  articles  and  reports  discarding  the  material 
that  did  not  address  directly  the  questions  of  priority-setting,  coordination,  or  measurement  of  R&D.  Some 
materials  that  were  discarded  included  articles  that  were  of  interest  to  the  general  category  of  R&D  but  that  were  not 
pertinent  to  the  focus  of  this  study,  e.g.,  an  article  debating  the  merits  of  the  federal  R&D  enterpnse  as  a  contributor 

to  economic  growth. 

As  this  material  was  being  collected,  we  developed  an  annotated  bibliography.  This  bibliography  was  shared 
at  several  points  with  the  NSB  staff  to  be  sure  that  we  were  identifying  the  most  relevant  literature.  We  amended 
our  search  strategies  based  upon  the  NSB  feedback,  by  adding,  for  example,  materials  about  performance  measures 
and  economic  returns  to  R&D  as  they  apply  to  priority  setting.  We  did  not  attempt  to  resolve  discrepancies  among 
these  sources.  We  relied  solely  on  the  literature  and  did  not  solicit  the  input  of  representatives  from  interested 
organizations,  such  as  the  NAS  or  AAAS  to  help  develop  a  set  of  priority  measures  across  fields  of  science. 

Finally,  in  November  1999,  we  submitted  a  draft  report  to  NSB  that  provided  the  results  of  the  literature 
review,  including  the  refined  models  for  priority  setting  across  fields  of  science,  recommendations  for  the  priority 
setting  measures,  and  a  method  for  verifying  data  submitted  in  the  annotated  bibliography.  Based  upon  subsequent 
conversations  and  the  comments  from  NSB  staff  on  the  earlier  document,  we  have  crafted  the  present  effort. 
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2.  INSTITUTIONS  AND  PRACTICES  OF  GOVERNMENT-WIDE 
COORDINATION  AND  PRIORITY  SETTING 


Goals,  priorities,  and  budget  allocation  are  all  a  part  of  a  pluralistic  system  for  creating  the  R&D  budget,  yet 
there  is  no  formally  defined  process  within  the  federal  government  to  set  goals  and  priorities  or  make  allocation 
decisions  for  science.  The  process  is  iterative  and  complex.  Many  players  with  different  interests  interact  to 
influence  the  outcomes.  As  the  National  Academy  of  Sciences  has  noted,  as  these  players  interact,  the  interests  of 
science  come  up  against  other  national  goals  and  priorities,  and  funding  for  specific  areas  of  science  compete  with 
one  anotiier  for  allocation  of  funds.  (NAS,  1995,  p.  4)  The  outcome  is  roughly  balanced  across  many  areas  of 
science  and  targets  a  number  of  specific  goals  and  priorities. 

This  section  assesses  the  discussion  in  the  literature  of  the  broad  goals  for  R&D  with  a  particular  focus  on 
the  government-wide  organizations  that  play  a  large  role  in  coordination  and  priority  setting  across  fields  of  science. 

SETTING  GOALS  -  OPERATIONAL  MODELS 

There  is  a  great  deal  of  attention  paid  in  the  available  literature  to  the  question  of  goal  setting  for  science. 
Most  of  the  reports  recommend  ways  that  goals  for  science  funding  could  be  better  set  within  the  system,  with  die 
implicit  criticism  that  the  current  political  process  no  longer  adequately  serves  the  people’s  interests  in  the  realm  of 
science.  The  issues  surrounding  goal  setting  for  science  have  taken  on  a  new  imperative  in  large  part  because  the 
science  and  technology  enterprise  has  entered  a  new  era  of  funding  challenges.  This  view  is  prevalent  among  most 
literature  a^/tr^cging  these  questions.  (Council  on  Competitiveness  1996,  Clinton  and  Gore  1994,  NAS  1993)  The 
challenges  inherent  in  this  new  era  arise  from  a  number  of  factors,  including  the  end  of  the  Cold  War,  the 
information  revolution,  and  the  increasing  use  of  science  and  technology  in  products  prominent  in  the  world 
economy.  “As  we  approach  the  new  milleimium,  these  broad  changes  have  recast  the  framework  in  which  the  U.S. 
research  and  development  system  functions.”  (NAS,  1993,  preface) 

Prominent  among  the  reports  recommending  a  more  rational  goals  process  for  science  funding  is  the 
National  Academy  of  Sciences  report.  National  Goals  for  a  New  Era,  which  recommends  that  1)  the  United  States 
should  be  among  the  world  leaders  in  all  major  areas  of  science;  and  2)  the  United  States  should  maintain  clear 
leadership  in  some  major  areas  of  science  (1993,  p.  19-20).  These  broad  goals  are  echoed  in  the  Clinton 
Administration  report,  Science  in  the  National  Interest,  which,  in  addition  to  adopting  the  NAS  recommendations, 
adds  the  goal  that  science  should  be  conducted  to  “enhance  coimection  between  fundamental  research  and  national 
goals.”  (Clinton  and  Gore,  1994,  p.  7) 

Among  the  reports  addressing  goal  setting,  recommendations  fall  into  three  broad  categories: 
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•  a  shift  fi-om  the  current  disaggregated  approach  to  one  tying  the  science  ftmding  to  national  goals  such  as 
health,  environmental  sustainability,  energy,  and  so  on;  (Clinton  and  Gore,  1994;  Carnegie,  December 

1992) 

•  a  scientific  goals  approach  which  advocates  cross-cutting  assessments  of  existing  government  spending 
in  areas  of  science  and  then  a  realignment  of  budgets,  if  needed,  to  better  meet  the  needs  of  scientific 
advancement;  (Merrill  and  McGeary,  1999)  and 

•  a  fine  tuning  of  the  existing  complex,  political  process.  (The  “Ehlers  Report,”  US  Congress,  1998) 

Criteria  reconunended  for  setting  goals  include  assessing  the  extent  to  which  science  will  help  meet 
important  goals,  and  the  standing  of  U.S.  science  relative  to  capabilities  in  the  rest  of  the  world.  Reports 
recommending  changes  to  the  goal  structure  for  science  are  mainly  made  in  reports  fi-om  the  National  Academy  of 
Sciences,  the  Carnegie  Commission,  and  the  Committee  for  Economic  Development.  Literature  developed  by 
groups  closer  to  the  process  are  less  likely  to  discuss  broad  goal  setting  for  science  and  more  likely  to  focus  on  the 
question  of  how  science  priorities  are  set  within  the  agencies. 

Several  reports  address  themselves  to  the  broadest  goal:  how  much  is  enough?  The  Office  of  Technology 
Assessment  (OTA)  notes  “Questions  such  as  ‘Does  the  Nation  need  more  science?’  and  ‘How  much  research  should 
die  Federal  Government  support?’  Have  no  ready  answers.  ‘How  much  is  enough’  depends  on  the  goals  of  the 
research  system”(OTA,  1991,  p.  10).  The  Clinton  Administration’s  report.  Science  in  the  National  Interest,  suggests 
that  “a  reasonable  long  term  goal  for  the  total  national  R&D  investment...might  be  about  3  percent  of  GDP,”  but  the 
report  does  not  offer  a  plan  as  to  how  to  reach  this  goal  (Clinton  and  Gore,  1994,  p.  15).  (During  the  Clinton 
Administration,  R&D  spending  as  a  share  of  GDP  has  actually  dropped.^  )  Leon  Lederman  suggests  that  tire  US 
should  be  spending  twice  the  amount  it  was  spending  in  1968  and  should  increase  spending  by  four  percent  per  year. 
(1991,  p.  6) 

Literature  focusing  on  goals-based  priority  setting  has  also  acknowledged  new  challenges  and  explored  new 
models.  This  literature  often  refers  back  to  larger  goals,  but  then  points  to  specific  disciplines  of  science  that  need 
attention.  Models  for  priority  setting  in  the  ideal  fall  into  three  broad  categories: 

•  Engaging  the  scientific  community  in  determining  priorities  based  on  scientific  needs,  linking  priorities 
to  upstream  and  downstream  connections  and  benefits.  (Carnegie  Commission,  1992) 

•  Benchmarking  U.S.  capabilities  in  specific  fields  against  those  of  other  nations  and  determining  where 
more  (but  rarely  less)  emphasis  might  be  placed  to  help  bolster  a  specific  area  of  science.  (NAS  1998; 

1993) 
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•  Having  agencies  seek  input  from  scientists  and  stakeholders  on  priorities  in  science  that  will  help  meet 
government  missions  like  space  science  or  energy  science.  (U.S.  Congress  1998) 

The  practice  of  priority  setting  at  the  agency  level  will  be  considered  in  section  three.  What  follows  is  an 
examination  of  what  the  literature  has  to  say  about  government-wide  players  and  the  processes  of  coordination  and 
priority  setting  at  the  supra-agency  level. 


EXECUTIVE  BRANCH  ORGANIZATIONS 

Scientific  disciplines  overlap  agency  jurisdictions,  making  attempts  to  coordinate  the  federal  R&D  portfolio 
more  difficult*  The  need  to  coordinate  Executive  Branch  R&D  activities  was  recognized  by  Congress  in  the 
National  Science  and  Technology  Policy,  Organization,  and  Priorities  Act  of  1976,  which  established  the  Office  of 
Science  and  Technology  Policy,  the  Presidents  Committee  of  Advisors  on  Science  and  Technology  (PCAST),  and 
the  Federal  Coordinating  Coimcil  for  Science,  Engineering,  and  Technology,  the  interagency  entity  that  was  the 
precursor  to  the  National  Science  and  Technology  Council  (NSTC).  This  structure,  together  with  the  operations  of 
die  Office  of  Management  and  Budget  (0MB),  the  bocfy  charged  with  coordinating  R&D  policy,  planning,  and 
funding  decisions  between  agencies.  However,  the  actual  operation  and  effectiveness  of  these  processes  in 
coordinating  R&D  policy,  planning,  and  funding  is  poorly  described  in  the  literature. 

The  Office  of  Science  and  Technolc^  Policy 

The  Office  of  Science  and  Technology  Policy  (OSTP)  in  the  Executive  Office  of  the  President,  under  the 
direction  of  the  Assistant  to  the  President  for  Science  and  Technology,  provides  a  resource  for  scientific  and 
tortiTiiral  information  withoi  the  government  and  plays  a  role  in  coordinating  R&D  activities  throughout  die 
agencies  and  ensuring  comprehensiveness  in  the  budget  for  science  activities.  The  science  advisor  and  his  staff 
serve  as  advisors  to  the  President,  and  so  act  to  implement  the  President’s  policy.  (OTA,  1991,  p.  74)  “OSTP  has 
the  access  to  outside  expertise  and  the  focus  on  sciences,  but  its  role  in  the  budget  process  is  subordinate  to  OMB.” 

(Teich,  1991) 

OSTP  helps  to  coordinate  federal  science  activities  to  meet  die  President’s  goals.  This  is  primarily  done,  in 
the  Clinton  Administration,  through  the  National  Science  and  Technology  Council  (NSTC)  for  which  OSTP  acts  as 
a  secretariat.  The  Executive  Office  of  the  President,  through  the  science  advisor  and  NSTC,  has  developed 


4  U.S.  Gross  Domestic  Expenditure  on  R&D  (GERD)  as  a  percentage  of  Gross  Domestic  Product  (GDP)  has  decreased 
from  2.74  percent  in  1992  to  2.64  percent  in  1997.  Source;  OECD’s  Main  Science  and  Technology  Indicators  1998,  volume  2, 
table  5, 

5  This  section  relies  heavily  on  the  1991  Office  of  Technology  Assessment  report,  “Federally  Funded  Research: 
Decisions  for  a  Decade,”  which  is  by  far  the  most  comprehensive  source  of  information  on  the  roles  of  the  different  parts  of 
government  in  setting  goal  and  priorities,  in  allocating  funds,  and  in  coordinating  research  activities. 
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mechanisms  to  review  specific  areas  (such  as  environment  and  natural  resources,  information  and  communication, 
or  transportation  R&D). 


Joint  OSTP-OMB  Priority  Memos 

OSTP,  together  with  the  Office  of  Management  and  Budget,  issues  a  budget  memorandum  to  the  agencies 
each  year  on  R&D  priorities.  The  guidance  provided  is  developed  dtrough  the  NSTC  process,  which  includes 
representation  from  all  the  R&D  agencies.  The  priorities  are  established  based  on  a  set  of  goals  established  by  the 
Clinton  Administration  in  the  first  months  of  its  tenure.  These  goals  included:  (1)  a  healthy,  educated  citizenry,  (2) 
job  creation  and  economic  growth,  (3)  world  leadership  in  science,  mathematics,  and  engineering,  (4)  improved 
environmental  quality;  (5)  harnessing  information  technology;  and  (6)  enhanced  national  security.  In  as  much  as 
there  is  no  clear  statement  as  to  how  priorities  are  to  be  derived  fi-om  these  goals,  it  is  useful  to  look  at  recent  budget 
memoranda  for  insights  into  this  process.  (Gibbons  and  Panetta,  1994) 

The  Administration  uses  a  memo  issued  jointly  by  the  OSTP  and  the  0MB  to  help  set  priorities  in  science 
and  technology.  Table  2-1  shows  the  evolution  of  research  priorities  named  by  the  Clinton  Administration. 
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Table  2-1:  Changing  Priorities  for  R&D  within  the  Clinton  Administration 

FY96 _ FY97 _ FY98 _ FY99 _ R^OO _ FY01 

Partnership  for  a  new  generation  of  vehicle _ _ _ 

Biomedical  research,  health  promotion  and  disease  and  injury  prevention  research _ 

Learning  and  copnitive  processes  _  — 

Materials  technology  _ ^ - - - - - 

Energy  production  and  utilization  technologies _ _ _ _ — 

Integrated  ecosystem  management  _ _ _ _ _ 

Networking  and  communications _ _ _ 

Human-computer  interaction  _ _ _ _ _ 

Counter-terrorism _ ; _ _ _ _ _ 

Detection,  monitoring,  and  verification _ _ _ 

Ciyilian  aircraft _ _ 

Telemedicine _ _ _ _ 

Infectious  diseases _ _ _ _ _ 

Environmental  risk  assessment 

Microelectronics  _ _ _ _ 

I  U.S.  Global  Climate  Change  Research  _ 

I  Plant  genome  research 

[Nanotechnology 


As  part  of  the  budget  process,  the  FY1996  budget  memo  says  that  die  NSTC  Committees  “will  work  with 
OSTP  and  OMB  using  Circular  A-1 1  ’s  R&D  Exhibits  to  collect  data  on  the  programs  that  contribute  to  the 
identified  priorities.”  The  memo  notes  that  “Continual  efforts  to  stimulate  and  coordinate  the  integration  of  cross¬ 
agency  initiatives”  will  be  undertaken.  (Gibbons  and  Panetta,  1994;  p.  15)  In  its  instructions,  the  two  offices  also 
call  for  a  series  of  “crosscuts”  in  education,  transportation,  construction,  computing,  environment,  and  fundamental 
science.  (Gibbons  and  Rivlin,  1995;  p.  10) 


Crosscuts 

In  addition  to  naming  research  priorities,  the  joint  memos  also  ask  the  agencies  for  information  on 
“Crosscutting”  themes  in  S&T.  The  FY  2000  joint  OSTP-OMB  budget  call  naming  priorities  for  R&D  notes  that 

“NSTC  coordinated  selected  interagency  science  and  technology  investment  priorities . managed  as  interagency 

crosscuts.”  (Jones  and  Lew,  1998;  p  2.)  The  crosscuts  listed  in  the  FY  2000  memo  are  limited  to  four: 

•  Climate  Change  Technology  Initiative 

•  U.S.  Global  Climate  Change  Research 

•  Partnership  for  New  General  of  V ehicles 

•  High  Performance  Computing  and  Communications. 

In  the  OSTP-OMB  joint  memo  naming  priorities  for  FY2001,  only  two  formal  crosscuts  are  included  in  the 
budget  call: 


Infonnation  technology  R&D,  and 
U.S.  Global  Climate  Change  Research. 
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President’s  Council  of  Advisors  on  Science  and  Technology  (PCAST) 

PCAST  advises  the  President  through  the  Assistant  to  the  President  for  Science  and  Technology.  Beyond 
that,  according  to  the  Executive  Order  12882  defining  the  role  for  PCAST,  its  principle  task  is  to  assist  the  NSTC  in 
securing  private  sector  involvement  in  the  latter’s  activities.  (EOP(a),  1993)  In  its  ovra  Statement  of  Principles, 
PCAST  emphasizes  the  need  for  strong  support  of  both  basic  and  applied  research  and  presses  for  funding  stability 
to  sustain  multi-year  efforts.  (PCAST,  no  date)  Some  of  the  recommendations  are  general  in  nature;  for  example,  in 
a  1996  letter  to  the  President,  PCAST  outlined  their  view  of  major  research  priorities  that  shotild  instruct  fimHing  for 
the  4-year  span  of  the  Administration’s  term.  (PCAST,  1996)  Beyond  this,  PCAST  has  assembled  review  panels 
and  ordered  studies  to  examine  specific  issues,  such  as  the  PCAST  Fusion  Review  Panel  report  on  the  U.S.  program 
of  fusion  energy  R&D,  and  made  recommendations  therein  about  specific  areas  of  research.  (PCAST,  1995) 

PCAST  also  makes  recommendations  based  on  its  own  reviews,  fi’om  technical  and  budgetaiy  perspective, 
of  reports  of  the  NSTC.  The  National  Nanotechnology  Initiative  is  an  example  of  recent  PCAST  influence  on  the 
priority  setting  process:  PCAST  studied  the  possibilities  for  nanotechnology  and  incorporated  its  findings  in  a  letter 
to  the  President  recommending  nanotechnology  as  a  new  priority.  (PCAST,  1999)  This  was  apparently  an  important 
step  in  getting  this  cross-agency  and  cross-field  initiative  underway.  (See  Table  2.1) 


The  National  Science  and  Technology  Council 


The  National  Science  and  Technology  Council  was  established,  according  to  the  Executive  Order  (12881) 

creating  it,  to  coordinate  the  S&T  policymaking  process  and  to  “integrate  the  President’s  S&T  policy  agenda  across 

the  Federal  Government.”  Further,  the  Order  states  that  NSTC  should  “ensure  S&T  are  considered  in  development 

and  implementation  of  federal  policies  and  programs.”  (EOP(b),  1993) 

All  executive  department  [sic]  and  agencies...shall  coordinate  S&T  policy  through  the  coimcil  and  shall 
share  information  on  R&D  budget  requests....The  Council  shall  develop  for  submission  to  the  Director 
of  die  Office  of  Management  and  Budget  recommendations  on  R&D  budgets  that  reflect  national  goals. 
In  addition,  the  Council  shall  provide  advice  to  the  Director  of  the  Office  of  Management  and  Budget 
concerning  the  agencies’  R&D  budget  submissions.  (EOP(b),  1993) 

This  purpose  ofNSTC  is  made  clearer  in  its’  1998  Annual  Report. 

Federal  departments  and  agencies  have  identified  a  set  of  R&D  areas  that  are  important  national  efforts 
requiring  coordinated  investments  across  several  agencies...This  budget  guidance,  rather  than  providing 
an  exhaustive  list  of  all  Administration  R&D  priorities,  focuses  on  those  activities  that  require 
significant  level  of  interagency  coordination.  (NSTC,  1998) 

Thus,  it  is  the  exceptional  R&D  initiative  that  is  coordinated  throughout  the  government  by  NSTC  and  it  is 
the  agencies  themselves  within  the  fi-ameworic  ofNSTC  rather  tiian  NSTC  as  a  separate  supervisory  entity  that  cany 
forward  this  work. 
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Available  government  documents  indicate  that  the  NSTC  takes  an  active  role  in  priority  setting  as  well  as 
coordination.  According  to  the  April  22, 1999  Lane-Lew  memorandum,  the  NSTC  convenes  during  the  budget 
process  to  “review  the  S&T  investment  portfolio  and  help  ensure  the  strongest  possible  R&D  budget  proposal  for 
the  [fiscal  year].”  (p.  1)  In  addition,  the  memo  notes  that  the  NSTC  “coordinates  a  small  number  of  selected 
interagency  science  and  technology  investment  priorities.”  For  example,  in  preparation  for  fiscal  year  2001,  the 
NSTC  is  overseeing  the  coordination  and  priority-setting  for  1 1  areas.  Of  these  1 1  areas,  the  more  mature, 
congressionally  mandated  programs  are  managed  as  formal  interagency  crosscuts,  while  areas  being  developed  for 
priority  attention  become  the  subject  of  NSTC  working  groups  designed  to  develop  programs.  The  NSTCprograms 
include  a  definition  of  activities,  an  inventory  of  existing  budgeted  research,  and  an  implementation  plan  for 

coordination  and  allocation  (Lane  and  Lew,  1999,  p.  2). 

NSTC  is  organized  as  a  series  of  standing  committees  covering  broad  aspects  of  the  portfolio  defined  by 
national  goals  for  S&T  who,  in  turn,  may  form  working  groups  (task  forces,  programs)  to  address  particular  issues. 
As  an  example,  the  NSTC  Committee  on  Science  has  subcommittees  addressing  a  variety  of  substantive  issues  as 
well  as  those,  such  as  the  subcommittee  on  biotechnology  which  “addresses  research  opportunities  in  aspects  of 
biotechnology,  resources  and  infinstructure  needed  for  biotechnology  research,  and  international  aspects  of 
biotechnology.”  (NSTC,  1998;  p.  23) 

The  role  of  the  NSTC  is  cited  in  several  reports  as  playing  a  key  role  in  priority  setting  and  coordination. 
David  Z.  Robinson  notes  feat  the  Executive  Office  of  the  President,  through  the  president’s  science  advisor  and  the 
NSTC,  “has  developed  mechanisms  to  review  specific  areas  (such  as  environment  and  natural  resources, 
infonnation  and  communication,  or  transportation  R&D).  Where  duplication  of  effort  and  gaps  can  be  identified, 
trade-offs  can  be  made.  These  crosscuts  are  indeed  useful  and  have  led  to  improved  efficiency.  Trade-  offs  are 
made  among  compatible  activities  and  they  make  sense”  (Robinson,  1997;  pp.  218-19)  Robinson  does  not  detail 
how  these  tradeoffs  are  conducted. 

The  Office  of  Management  and  Budget 

The  Office  of  Management  and  Budget  compiles  the  President’s  annual  budget,  seeking  within  constraints 
to  support  the  President’s  policies,  programs,  and  commitments.  The  formal  process  starts  each  summer  in  the 
Executive  Branch  when  the  agencies  begin  preparing  their  budgets  for  the  fiscal  year  beginning  in  October  of  the 
following  year.  Each  summer,  budget  guidelines  for  R&D  are  issued  jointly  by  OSTP  and  0MB.  These  guidelines, 
discussed  above,  work  more  as  a  budget  call,  asking  agencies  to  report  on  their  plans  to  conduct  R&D  in  specific 
priority  areas.  0MB  reviews,  coordinates,  and  scrubs  the  budget,  resolves  any  disagreements,  and  submits  the 
budget  to  Congress  in  late  January  or  early  February.  There  is  no  “R&D  budget”  as  such;  there  are  many  R&D 
budgets  embedded  in  the  budgets  of  21  federal  departments  and  agencies  that  conduct  or  support  R&D.  The  R&D 
requests  are  handled  in  the  budget  process  as  parts  of  the  agency’s  total  budget.  (Shapely,  1992.) 

0MB  has  die  power  to  shape  the  budget,  but  ensuring  that  the  budget  responds  to  the  President’s  priorities 
for  research  are  only  a  small  part  of  its  responsibilities.  The  0MB  has  responsibility  to  examine  agency  proposals 
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for  redundancy  and  looks  for  opportunities  for  interagency  coordination.  Although  the  (pre-NSTC)  OTA  study  cited 

0MB  as  “a  surrogate  for  a  crosscutting  agenf’  within  the  federal  system,  (OTA,  1991,  p.  14),  priority  setting  across 

different  areas  of  science  or  among  the  different  R&D  budgets  does  not  occur  at  0MB: 

Setting  priorities  implies  a  ‘trade-off  area  composed  of  the  programs  to  be  prioritized  and  some  agreed- 
on  criteria  for  deciding  priorities.  Priorities  may  meaningfully  be  set  for  programs  having  the  same  or 
similar  goals.  But  between  programs  for  independent  goals,  or  when  the  programs  have  multiple  goals, 
priority  setting  is  essentially  arbitrary.  The  major  federal  R&D  programs  have  different  goals,  and  the 
major  budgetary  decisions  on  them  are  largely  independent  of  each  other  (Shapley,  1992,  p.  10). 


Although  a  munber  of  reports  and  analyses  call  for  0MB  and  others  within  the  budget  process  to  evaluate 

the  R&D  portfolio  in  terms  of  its  progress  towards  national  goals,  in  Shapley’ s  view,  efforts  at  priority  setting  for 

R&D  should  not  involve  trade-offs  within  the  budgets  dedicated  to  R&D  spending.  These  are  essentially  political 

judgments,  he  claims,  and  need  to  be  placed  within  the  total  budgetary  process: 

R&D  budgets  should  be  considered  primarily  in  relation  to  their  program  goals,  not  in  relation  to  R&D 
for  other  goals.  The  relative  priorities  of  the  many  independent  program  goals  must  be  set  arbitrarily  in 
the  political  process....There  is  no  reason  why  R&D  programs  or  major  initiatives  directed  at  different 
goals...should  be  expected  or  required  to  compete  only  with  each  other  in  the  budget  process.  These  and 
the  goals  they  are  to  serve  should  compete  with  all  other  proposals  in  the  federal  budget  (Shqrley,  1992, 
p.ll). 

0MB  expects  the  scientitic  community  to  work  with  tirem  on  these  priority  areas  by  helping  to:l)  identify 
the  implications  and  priorities  of  all  the  programs  that  the  0MB  is  trying  to  put  together;  2)  explain  the  programs,  to 
scientists.  Congress,  and  others,  3)  reinvent  programs  when  necessary.  (Glauthier,  1995) 


CONGRESS 

Of  all  the  institutions  involved  in  coordination  and  priority  setting  across  fields  of  science,  the  literature  as 
a  whole  shows  its  largest  g^  in  its  treatment  of  Congress.  As  the  source  of  all  the  resources  to  be  used  in  building 
and  sustaining  the  federal  research  portfolio  its  role  is  crucial,  yet  its  methods  and  processes  appear  to  have  been  less 
amenable  to  exposition  than  in  the  case  of  other  institutions. 

While  having  provided  the  Executive  Branch  with  a  means  to  coordinate  R&D  through  the  1976  Science 
Policy  Act,  Congress  has  not  made  a  parallel  effort  to  coordinate  its  own  R&D  policy,  planning,  and  funding  efforts. 
At  least  21  Congressional  Committees  have  direct  federal  R&D  policy  or  funding  responsibility  (OTA,  1991 ;  p. 

83)®.  Nine  of  the  thirteen  appropriations  subcommittees  have  R&D  responsibilities  (Carnegie  Commission,  1994). 

In  addition,  the  committee  structure  and  jurisdictional  boundaries  between  tiie  House  of  Representatives  and  Senate 
are  different,  further  complicating  comprehensive  policy  and  budget  coordination.  (OTA,  1991;  Shapely,  1992; 
Carnegie,  1994) 


®  Note  that  this  figure  results  fi^om  adding  the  two  budget  committees  and  the  two  appropriations  committees  to  the 
totals  reported  in  the  OTA  study. 
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At  no  time  in  the  Congressional  authorization  or  appropriations  process  is  there  a  comprehensive  view 
taken  of  the  R&D  portfolio  across  the  federal  government.  While  numerous  studies  have  cited  the  need  for  this 
coordination.  Congress  has  not  responded  and  a  major  shortcoming  still  exists.  (NRC,  1993;  NAS  1988;  Carnegie 
Commission  1993,  among  others) 

Other  structural  and  legal  impediments  exist  to  impede  comprehensive  coordination  of  R&D  policy  and 
budgets  within  Congress.  For  most  of  the  last  decade,  the  budgets  of  the  defense  functions  and  the  civilian  functions 
have  been  kept  separate  by  budget  “firewalls”  enacted  after  the  end  of  the  Cold  War  to  prevent  the  raiding  of  the 
defense  budget  to  pay  for  civilian  program  increases.  With  over  half  of  the  federal  R&D  budget  in  the  defense 
programs  (budget  function  #  050),  these  “firewalls”  effectively  thwart  efforts  to  coordinate  R&D  policy,  planning, 
and  fimding  within  Congress.  (Carnegie,  1994)  So  at  the  Congressional  level,  for  example,  coordinating  the  Navy 
oceanographic  budget  with  the  oceanographic  budget  at  NSF  and  NOAA  would  be  difficult. 

Another  impediment  to  adequate  coordination  in  Congress  has  been  the  reduction  in  the  role  that 
Congressional  support  agencies  play  in  advising  Congress  on  R&D  policy,  planning,  and  funding  issues.  Of 
primary  importance  was  the  termination  of  the  Congressional  Office  of  Technology  Assessment  (OTA).  More 
recently,  the  Congressional  Research  Service  (CRS)  merged  its  Science  Division  into  another  division,  effectively 
downgrading  the  R&D  program  analysis  functions.  Reports  have  cited  the  role  of  Congressional  support  agencies  in 
helping  Congress  (Carnegie  Commission,  1991;  1994)  and  urged  a  strengthening  of  their  role  in  providing  expert 
advice  to  legislators.  Given  these  earlier  findings,  the  elimination  of  the  OTA  and  changes  at  CRS  can  be  viewed  as 
making  more  effective  coordination  of  R&D  policy  more  difficult. 

In  addition  to  the  direct  funding  decisions  made  by  Congress  there  are  a  range  of  other  legislative  decisions 
that  can  affect  planning  and  priorities  of  federal  agencies  and  the  conduct  of  federally-funded  R&D.  Regulatoty 
decisions  (disposal  of  toxic  chemicals  used  in  research,  treatment  of  laboratory  animals),  tax  decisions  (R&D  tax 
credit  extension),  or  other  decisions  affecting  research  institutions  (immigration  visa  ceilings,  graduate  medical 
education  subsidies)  can  all  affect  R&D  programs.  Yet  tiiese  decisions  are  made  outside  of  the  Congressional 
circles  in  which  R&D  policy  is  decided  and  are  frequently  not  coordinated  with  the  Congressional  entities  having 
R&D  policy  jurisdiction.  (Carnegie,  1994.) 

Interactions  Between  the  Executive  and  Legislative  Branches  on  R&D 

In  addition  to  the  processes  within  each  of  the  Executive  and  Legislative  branches  of  government,  decisions 
on  agency  funding  and  operations  involve  an  interaction  between  the  branches  as  well.  The  Executive  Branch 
proposes  budgets,  agency  operational  plans  and  GPRA  reviews  and  Congress  acts  on  those  recommendations  and 
reports.  Because  of  the  need  for  this  interaction,  the  shortcomings  found  within  each  branch  of  government  become 
compounded  by  the  lack  of  a  formal  coordinating  mechanism  between  the  Branches. 

Congressional  interaction  with  the  Executive  Branch  on  R&D  issues  occurs  in  a  variety  of  ways.  The  most 
visible  forms  occur  during  the  budget  and  appropriations  process  each  year,  since  R&D  functions  are  funded  out  of 
the  discretionary  budget  and  are  thus  subject  to  annual  appropriations.  But  as  has  been  noted  above  there  are 
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limitations  on  both  ends  of  Pennsylvania  Avenue  that  prevent  this  interaction  from  accomplishing  the  level  of 
coordination  between  the  branches  of  government  that  many  participants  express  as  their  desire. 

In  addition,  any  formal  interactions  are  somewhat  stilted  and  confined.  Since  0MB  must  approve  agency 
testimony,  any  formal  presentation  to  Congress  will  serve  to  ratify  the  final  decisions  made  during  the  budget 
submission  and  will  not  easily  provide  a  vehicle  for  an  R&D  agency  to  comment  critically  upon  decisions  made  in 
that  process.  In  addition,  agencies  are  generally  nonresponsive  to  questions  of  priority  put  to  them  by  Congress, 
since  any  admission  of  a  lower  priority  for  a  program  has  usually  resulted  in  a  funding  cut,  especially  during  the 
deficit  reduction  efforts  that  have  marked  the  last  decade.  (Shapely,  1992) 

One  tool  that  has  been  increasingly  used  by  Congress  since  the  mid-1980’s  to  communicate  its  priorities  for 
federal  spending  is  “earmarking.”  This  process  of  designating  institution-specific  funding  has  been  discussed 
widely  as  a  violation  of  the  peer-review  process  of  selecting  research  priorities  (Carnegie,  1994).  But  this  does 
constitute  a  fi^quently  used  means  of  communicating  R&D  funding  priorities  between  the  Congress  and  the 
Executive  Branch  agencies. 

Separate  from  the  appropriations  process  is  the  role  that  the  authorizing  committees  play  in  conducting 
reviews  of  federal  programs.  In  tiie  R&D  area.  Congressional  oversight  is  the  systematic  review  of  programs  by 
authorizing  committees  that  seek  to  address  the  problems  of  research  management  and  priority  setting  outside  of  the 
annual  appropriations  process.  These  efforts  have  been  important  in  shaping  the  direction  and  funding  of  federal 
programs.  Problems  exposed  during  oversight  hearings  on  the  Superconducting  Supercollider  were  ultimately 
responsible  for  its  cancellation.  These  oversight  hearings  play  a  key  role  in  determining  the  budgets  of  specific 
research  programs  and  encouraging  coordination  between  the  research  agencies.  (OTA,  1991) 

But  while  studies  have  sought  improved  coordination  of  R&D  policy,  planning,  and  funding  between  the  two 
branches  of  government,  as  mentioned  above,  no  formal  mechanism  has  emerged.  Absent  a  formal  agreement,  the 
budget,  appropriations,  and  oversight  processes  constitute  the  coordination  mechanisms  and  the  conduct  of  this  work 
is  subject  to  individual  agency  and  committee  dynamics  and  is  frequently  left  to  the  perspectives  and  proclivities  of 
individual  Members  of  Congress  and  their  staffs. 
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3.  PRIORITY  SETTING:  AGENCY  PERSPECTIVES  AND  CASES 


This  section  presents  the  description  of  alternative  models  for  priority  setting  in  different  agencies  as 
revealed  in  the  literature.  To  provide  further  illumination,  albeit  from  a  different  angle,  this  section  also  presents 
cases,  as  described  in  the  literature,  of  how  some  specific  cross-agency  research  initiatives  were  initiated. 

PARALLEL  ASSESSMENTS  FOR  DIFFERENT  AGENCIES  INVOLVED  IN  PRIORITIZING  R&D 

The  literature  on  agency-level  R&D  priority-setting  is  not  a  robust  collection.  The  Office  of  Technology 
Assessment  report  “Federally  Funded  Research:  Decisions  for  a  Decade”  describes  these  activities,  and,  even  though 
the  report  is  now  10  years  old,  this  section  draws  heavily  from  that  report  because  no  report  has  updated  it.  The 
agencies  themselves  have  issued  GPRA-inspired  strategic  plans  that  provide  some  insights  into  the  priority  setting 

process. 

Outside  of  these  sources,  we  found  very  little  on  what  goes  on  in  die  agencies  regarding  priority  setting, 
despite  the  fact  that  there  are  over  20  government  agencies  spend  R&D  money.  It  makes  sense  that  die  largest 
spenders  would  be  the  most  well  represented  in  the  literature,  but  smaller  agencies  most  likely  make  dearer  trade¬ 
offs  in  funding,  and  thus  may  be  more  interesting  to  stuity.  Table  3-1  below  summarizes  what  we  found  in  the 
literature  about  agency  priority  setting  activities  (the  full  text  description  is  in  Appendix  A.)  Not  all  of  these 
representations  may  be  up-to-date.  Some  of  the  literature  is  dated  and  changes  may  well  have  occurred  m  these 
agencies  since  the  original  report  was  written.. 

As  OTA  described  it,  “Within  agency  research  programs,  research  proposals  have  traditionally  been 
selected  for  support  on  the  basis  of  expert  peer  or  program  manager  judgments  of  scientific  merit  and  program 
relevance.”  (p.  96)  Moreover,  as  noted  in  Section  2  above,  each  agency  defines  R&D  somewhat  differently 

depending  upon  its  mission.  As  the  OTA  study  noted: 

“Each  agency  has  its  own  culture,  which  contributes  not  only  to  its  success,  but  also  embodies 
historically  the  “way  things  are  done.”  Agency  culture  is  thus  a  powerful  determinant  of  future 
directions,  with  specific  goals  reflected  in  the  collective  knowledge  of  agency  personnel.  Pluralism  and 
decentrali^tion  characterize  each  of  the  research  agencies,  with  many  separate  programs  pursuing 
diverse  objectives.  In  particular,  the  lines  of  decisionmaking  within  an  agency  are  more  complicated 
than  any  organizational  chart  would  suggest.”  (p.  97) 

As  Christopher  Hill  has  written,  “R&D  spending  responds  to  national  crises;  political  and  budgetary 
imperatives;  and  occasionally,  to  determined  efforts  of  highly  committed  interests  pursuing  long-term  visions  of  a 
different  future.”  (Hill,  1995)  No  single  model  of  decisionmaking  exists  to  illustrate  tiiese  mechanisms.  Although 
fliere  are  some  similarities  in  approach,  each  agency  has  a  different  set  of  decisionmaking  processes  to  aid  priority 

setting. 


AGENCY  (IN  PRINCIPLE  FINDINGS  METHODS  USED  TO  IDENTIFY  SPECIAL  OR  UNIQUE  GAPS  IN  SOURCES 

ORDER  OF  ABOUT  PRIORITY  PRIORITIES  METHODS  USED  WITHIN  LITERATUR  OF 
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_ agency^s  special  attention. _ _ _ 

NASA  NASA  sets  priorities  in  Priority  setting  results  from  a  In  the  1997  Strategic  Plan,  the  Not  clear  from  OTA;  NASA 

conjunction  with  the  budget  combination  of  input  from  NASA’s  own  OSS  proposed  near-term  (2000-  literature  how  Strategic 

process  and  by  selecting  specific  internal  managers,  staff  and  directors,  2004)  and  long-term  (2005-  NASA  decides  Plans;  GPRA 

projects.  Influenced  more _ and  external  actors  like  the  National _ 2020)  priorities.  The  plan  on  specific _ documents: 


heavily  by  Congress  than  other  Research  Council  the  Task  Group  on  established  seven  fundamental  missions  or  on  National 

agencies.  The  process  is  Space  Astronomy  and  Astrophysics,  and  questions  that  form  the  how  Research 

essentially  bottom  up  with  the  space  science  community.  In  its  goal  foundation  for  the  office’s  disinvestments  Council 

project  managers  proposing  new  to  pursue  scientific  excellence,  the  scientiHc  program.  It  then  are  made.  reports 

initiatives.  When  large  missions  Office  of  Science  emphasized  the  use  of  incorporated  the  advice  of  the 


groups  on  program  issues.  Goals  understanding  in  its  own  field  and 
are  set  “by  scientific  opportunity  across  fields?  Is  it  creative  and 
and  the  proposal  process,  as  well  original? 

as  in  special  initiatives  from  •  What  are  the  broader  impacts  of 

advisory  panels.”  the  proposed  activity?  E.g.,  advance 
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group  of  engineering  and  science 
advisors)  and  expert  panels 
convened  by  NAPA  and  the 
National  Academy  of  Sciences 
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CASES  OF  INTER-AGENCY  PRIORITY  SETTING  ACROSS  FIELDS  OF  SCIENCE 

This  subsection  briefly  describes  several  representative  cases  of  research  initiatives  to  illustrate 
priority  setting  found  in  the  literature.  The  literature  is  not  sufficient  to  provide  all  one  would  wish  to  know 
about  cross-agency  priority  setting  across  fields  of  science.  This  is  both  due  to  a  paucity  of  documentation 
and  to  the  fact  that  actual  initiatives  may  come  about  outside  of  the  channels  and  procedures  the  literature 
describes.  Anecdotal  cases  gleaned  from  the  literature,  those  that  have  not  been  formally  produced  as  case 
studies,  generally  are  characterized  by  the  difficulty  of  capturing  the  crucial  “pre-histoiy”  of  the  initiative, 
that  is  die  period  before  it  entered  the  formal  process. 

Partnership  for  a  New  Generation  of  Vehicles 

The  Clinton  Administration’s  Partnership  for  a  New  Generation  of  Vehicles  (PNGV)  which  began 
its  existence  as  the  so-called  Clean  Car  Initiative,  is  an  example  of  a  multi-million  dollar,  interagency  R&D 
effort  whose  origin  lies  largely  outside  the  fonnal  system  for  coordination.  It  is  true  that  the  initiative  can 
trace  its  lineage  back  to  flie  FCCSET  in  the  latter  days  of  the  Bush  Administration.  That  body  formed  the 
Advanced  Manufacturing  Technology  Initiative.  Arising  from  then  prevalent  concerns  over  U.S.  industrial 
and  technological  competitiveness,  it  sought  to  promote  synergy  in  manufacturing  R&D  in  general 
throughout  the  government. 

In  the  early  days  of  the  Clinton  Administration,  an  inter-agency  working  group  began  to  look  at  the 
automotive  industry.  This  group  began  to  outline  what  technology  goals  might  be  feasible  and  desirable  for 
this  industry  in  particular. 

Another  thread  was  the  contemporaneous  concern  over  the  fate  and  mission  of  the  national 
laboratories.  After  the  end  of  the  Cold  War,  there  was  a  general,  unfocused  interest  in  drawing  upon  the 
talents  and  output  of  the  labs  to  further  a  wider  national  goals  agenda.  One  means  for  doing  so  was  to 
allow,  through  the  establishment  of  cooperative  R&D  agreements  (CRADAs),  national  laboratory 
persormel  and  working  groups  to  work  with  and  for  private  industry  on  well-defined  research  projects. 

Nevertheless,  PNGV  came  about  as  a  political  directive  outside  of  the  fonnal  R&D  coordination 
^paratus  as  it  then  existed.  “Word  came  down... that  ‘the  President  wants  to  do  a  clean  car’”  (Chapman, 
1998).  The  initiative  was  not  in  the  final  budget  that  was  given  by  0MB  to  the  White  House  in  1995  after 
months  of  deliberations  by  the  agencies.  Rather,  it  appeared  at  the  end  of  the  process,  when  the  budget  was 
announced.  At  the  time,  this  directive  resulted  in  a  great  deal  of  scurrying  about  by  the  agencies  to  figure 
out  what  the  initiative  was.  As  a  result  and  only  after  the  fact,  a  great  deal  of  existing  research  was 
redesignated  as  Clean  Car  work  and  constituted  the  bulk  of  the  original  new  initiative. 
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National  Nanotechnology  Initiative 

The  National  Nanotechnology  Initiative  is  the  most  recent  of  the  large,  visible  cross-agency 
research  efforts  inaugurated  by  die  current  apparatus  for  coordination  and  priority  setting.  NSTC 
commissioned  the  creation  of  an  Interagency  Working  Group  on  Nanotechnology  (IWGN).  IWGN  was  a 
task  force  composed  of  representatives  of  a  wide  range  of  research-sponsoring  federal  agencies  as  well  as 
those  for  whom  this  issue  area  is  of  potential  importance  in  consideration  of  their  agency  missions.  The 
IWGN  deliberated  and  issued  a  report  calling  for  a  federal  initiative  in  this  area  (NSTC,  2000).  PCAST 
then  made  recommendations  based  upon  is  own  review,  from  technical  and  budgetary  perspective,  of  the 
comprehensive  report  of  the  IWGN.  The  result  was  a  letter  to  the  President  stating  that  the  members  of 
PCAST  “believe  that  the  Administration  should  make  NNI  a  top  priority.”  (PCAST  Letter,  November 
1999)  The  President  then  made  formal  armouncement  of  the  NNI  in  January  2000  at  CalTech. 

In  its  review,  PCAST  identified  the  federal  role  as  being  appropriate  due  to  the  present  distance  of 
the  relevant  science  (and,  indeed,  a  great  share  of  the  technology)  from  most  commercial  applications.  The 
members  in  their  review  also  draw  attention  to  the  inter-disciplinarity  required  to  pursue  the  R&D  in  this 
field.  They  note,  in  their  capacity  as  a  bridge  to  business,  that  the  effort  required  would  be  "incompatible 
with  many  current  corporate  structures.”  (PCAST  Review,  November  1999): 

There  Is  a  clear  need  for  Federal  support  at  this  time.  Appropriately,  Federal  and  academic  investments 
in  nanotechnology  R&D  to  date  have  evolved  in  open  competition  with  other  research  topics,  resulting 
in  some  fi:agmentation  and  duplication  of  efForts...Nanotechnology  will  require  a  somewhat  more 
coherent,  sustained  investment  in  long-term  research.  The  NNI  would  support  critical  segments  of  this 
research  and  increase  the  national  infiastructure  necessary  to  conduct  it.  (op.  cit.,  p.2) 

The  initiative  is  judged  fevorably  in  providing  balanced  funding  across  “fundamental  research, 
grand  challenges,  centers  and  networks  of  excellence,  research  infrastructure,  and  education  and  training.” 

As  a  result  of  these  factors,  the  document  designates  this  as  a  field  of  endeavor  requiring  multi-agency 
effort  and  su^ests  that  the  NRC  would  be  an  appropriate  external  body  to  provide  an  annual  review  of 
performance  under  the  NNI.  Interestingly,  PCAST  also  noted  that  changes  in  technology  this  profound 
could  potentially  lead  to  unforeseen  ethical  and  moral  questions.  In  their  review  fiiey  explicitly  suggest  that 
a  “modest”  amount  be  set  aside  for  the  study  of  such  implications  from  nanotechnology. 

AIDS  Research 

A  third  example  of  a  shift  in  R&D  priorities  is  the  rapid  increase  in  government  spending  on  AIDS 
research.  Lobbying  by  interest  groups  (such  as  the  Treatment  Action  Group  or  TAG)  had  an  affect  on 
Congressional  appropriations  for  health  research.  In  addition,  as  a  result  of  external  pressure,  the  National 
Institutes  of  Health  has  convened  several  study  groups  to  make  recommendations  explicitly  targeted  at 
AIDS  research.  In  order  to  help  prioritize  funding  in  the  area  of  AIDs-related  research,  the  NIH  Office  of 
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AIDS  Research  (OAR)  in  1994  formed  an  advisory  cormcil  to  help  identify  whether  AIDS  funding  should 
be  increased  and,  if  so,  in  what  areas  of  science  these  increases  should  be  made. 

The  final  report  of  the  Advisory  Council  to  the  OAR  makes  a  series  of  recommendations  designed 
to  further  enhance  the  NIH  AIDS  research  program  by  "creating  more  effective  coordination  of  the 
endeavor  and  identifies  several  key  scientific  areas  diat  warrant  special  emphasis"  (accompanying  letter, 
Charles  C.J.  Carpenter,  M.D.,  Chair  Office  of  AIDS  Research  Advisory  Council.)  The  report  recommends 
an  increased  emphasis  on  investigator-originated  research  and  a  reinvigoration  of  research  efforts  in 
vaccine  and  prevention  research.  The  report  makes  a  long  list  of  recommendations  for  AIDS  research, 
including: 

•  increase  support  for  and  improve  peer-review  of  investigator-initiated  research 

•  establish  a  restructured  trans-NIH  vaccine  research  effort 

•  augment  research  efforts  to  better  understand  the  human  immune  system 

•  develop  a  comprehensive  NIH  HIV  Prevention  Science  Agenda.  (Report  Of  The  NIH  AIDs 
Research  Program  Evaluation  Working  Group  Of  The  OfiSce  Of  AIDs  Research  Advisory 
Coimcil,  March  14, 1996.) 

The  report  had  some  influence  on  the  way  in  which  AIDS  research  funds  were  allocated,  according 
to  an  article  in  the  New  York  Times  (March  14  1996).  Nevertheless,  the  Executive  Office  did  not  enact 
one  of  the  principle  recommendations,  preserving  a  central  allocative  authority  for  AIDS  research  funding. 
A  number  of  administrative  changes  were  made  which  led  to  funds  for  AIDS  research  to  be  allocated 
among  various  agencies  rather  than  through  a  central  office: 

OAR  chief  William  Paul  was  informed  by  the  Office  of  Management  and  Budget  that  the  1996 
AIDS  budget  would  go  directly  to  each  NIH  institute.  OAR,  backed  by  the  Clinton 
Administration,  had  lobbied  hard  to  prevent  fliis,  contending  the  office  should  retain  the  budget 
authority  Congress  gave  it  3  years  ago.  OAR’s  independence  was  cast  into  doubt  by  a  legal 
technicality  that  Congress  included  in  the  January  resolution  that  funded  NIH  through  1996 
(Science,  19  January,  p.  281). 

Congress  did  so  despite  protests  from  prominent  scientists  and  AIDS  activists  that  the  move  would 
prevent  OAR  from  responding  to  a  massive  review  of  AIDS  research.  Although  OAR  didn’t  like  the 
situation,  it  urged  viewing  this  change  in  perspective  in  that  the  new  rule  applied  only  to  the  1996  budget, 
which  OAR  has  already  shaped.  Yet,  Congress's  action  was  viewed  as  "significant"  since  it  affected  OAR’s 
ability  to  make  changes  in  NIH’s  AIDS  portfolio  recommended  by  the  research  review.  "If  we  had  the 
authority  this  year,  we  would  have  used  it,"  says  Paul. .  (Science,  March  1, 1996) 


Astronomy  and  Astrophysics 

A  fourth  and  final  look  at  specific  cases  of  shifts  in  R&D  priority  setting  focuses  on  the  potential  influence 
of  a  scientific  community  on  the  spending  on  R&D  as  well  as  the  directions  for  science  and  technology  in  their  field. 

For  the  past  four  decades,  astronomical  and  astrophysical  communities  have  influenced  federal  spending  in 
their  field.  By  establishing  priorities,  and  reporting  them  to  relevant  government  agencies  and  the  U.S.  Congress, 
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the  National  Research  Council's  Astronomy  and  Astrophysics  Survey  Committee  has  had  direct  input  into  the 
priority  setting  process.  Beginning  in  1964  the  ^vork  of  the  Whitford  Committee,  this  scientific  community  has 
reported  each  decade  on  the  priorities  for  that  field  of  science  in  ways  that  have  charted  the  path  of  research  in 
astronomy  and  astrophysics.  (NAS  website) 

The  surveys  of  scientists  sponsored  by  the  National  Academy  of  Science  Committee  on  Astronomy  and 
Astrophysics  (CAA)  have  resulted  in  these  reports.  The  CAA  monitors  the  status  of  space  and  groimd  based 
astronomy  and  astrophysics  and  provides  assessments  to  the  National  Science  Foundation,  NASA,  and  other 
institutions  as  a  joint  committee  of  the  Space  Studies  Board  and  the  Board  on  Physics  and  Astronomy.  The  overall 
objective  of  the  CAA  is  to  encourage  progress  in  astronomy  and  to  assist  the  federal  government  in  program  choice 
and  implementation  in  these  fields.  (Mission  statement  of  the  CAA.) 

The  CAA  has  both  made  broad  recommendations  and  worked  specifically  with  NASA  to  establish 
priorities.  In  their  1997  report,  the  CAA  chose  among  the  many  options  and  identified  four  particularly  important 
and  timely  priorities  in  space  astrophysics  for  the  early  years  of  the  coming  decade.  In  ranked  order  (itself 
remarkably  unusual  even  among  so-called  priority-setting  processes)  the  recommended  priorities  were: 

1.  Determination  of  the  geometry  and  content  of  the  universe  by  measurement  of  the  fine-scale  anisotropy 
of  the  cosmic  microwave  background  radiation; 

2.  Investigation  of  galaxies  near  the  time  of  their  formation  at  very  high  redshift; 

3.  Detection  and  study  of  planets  around  nearby  stars;  and 

4.  Measurement  of  the  properties  of  black  holes  of  all  sizes. 

The  second  and  third  priorities  were  given  virtually  the  same  weight.  Four  additional  scientific  objectives 
were  judged  to  be  of  high  priority  but  nevertheless  less  urgent  than  the  primary  four  listed  above,  or  less  achievable 
in  terms  of  possible  space  missions.  These  recommended  secondary  objectives,  unranked,  are: 

•  Study  of  star  formation  by,  for  example,  high-resolution  far-infixed  and  submillimeter  observations  of 
protostars,  protoplanetary  disks,  and  outflows; 

•  Study  of  the  origin  and  evolution  of  the  elements; 

•  Resolution  of  the  mystery  of  the  cosmic  gamma-ray  bursts;  and 

•  Determination  of  the  amount,  distribution,  and  nature  of  dark  matter  in  the  universe  (A  New  Science 
Strategy  for  Space  Astronomy  and  Astrophysics,  Executive  Summary.) 

Other  efforts  by  this  committee  to  influence  priority  setting  have  included  a  letter  to  the  National  Science 
Foundation,  sent  at  that  agency's  request,  on  the  direction  and  usefulness  of  NSF's  facility  instrumentation  program 
(FIP).  In  this  letter,  the  committee  reaffirms  its  support  for  the  NSF  FIP  program,  but  suggests  program  changes  in 
the  length  of  telescope  time  available  to  the  public  and  other  modifications.  Other  committee  letters  have  been  sent 
on  Antarctic  astronomy,  cooperation  with  the  European  Space  Agency,  and  on  the  Cassini  Saturn  Probe.  (NAS 
website) 
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4.  CRITIQUES  AND  SUGGESTIONS  PRESENTED  IN  THE  LITERATURE 


Critiques  and  suggestions  in  the  literature  on  federal  R&D  budget  coordination  and  priority-setting  most 
often  take  the  form  of  normative  statements  at  a  fairly  high  level  of  generality.  Proposals  offering  procedural  details 
on  alternative  mechanisms,  discussing  how  systemic  transitions  might  be  staged  or  speculating  on  possible 
secondary  effects  from  such  shifts,  are  almost  non-existent.  Detailed  critiques  of  specific  methodological 
approaches  are  even  more  rare  than  pieces  detailing  methodologies  as  applied  in  agencies  or  other  bodies.  Most 
pieces  tend  to  advocate  some  selective  change,  rather  than  offer  an  integrated  design. 

What  follows  is  a  review  of  what  was  found  to  be  most  relevant  in  answering  the  questions  posed  by  the 
Committee.  It  will  be  noted  that  the  literature  rarely  addresses  these  questions  directly  and  so  must  be  culled  for 
passages  shedding  light  on  the  issues.  It  will  also  be  quite  clear  that  there  is  a  fine  line  between  critiques  of  current 
methodologies  and  implicit  suggestions  for  making  improvements  to  existing  process. 

CRITIQUES  OF  CURRENTLY  EMPLOYED  METHODOLOGY 

Perhaps  the  most  fi-equent  critique  addresses  a  perceived  lack  of  clear  methodology  for  priority  setting  and 
coordination.  This  is  implicit  in  the  first  Clinton  administration  statement  of  policy  in  this  area.  Science  in  the 
National  Interest  tasked  federal  agencies  as  follows: 

Each  agency  that  depends  on  or  contributes  to  our  science  and  technology  base  will,  with  involvement  of  the 
scientific  coiranunity,  delineate  its  fundamental  research  and  education  missions  with  respert  to  the  national 
goals;  develop  long-range  plans  for  its  fundamental  science,  mathematics,  and  engineering  investment;  and 
develop  measures  to  evaluate  its  contributions.  (Clinton  and  Gore,  1994;  p.  19) 

With  the  enactment  of  the  Government  Performance  and  Results  Act  (GPRA)  in  1993,  this  implicit  critique  has 
begun  to  be  addressed.  Yet,  a  furflier  implicit  critique  may  be  found  in  the  actions  of  the  House  Science  Committee 
which  held  hearings  in  1996  and  1997  on  implementation  in  the  civilian  science  agencies  and  announced,  in  1997, 
that  this  would  be  a  major  oversight  target.  Several  committees  asked  agencies  to  link  FY1999  and  FY2000  budget 
requests  to  goals  expressed  in  their  strategic  and  performance  plans.  (CRS  Reports  97-70, 97-382,  RS20257,  and  98- 

224) 

A  major  argument  in  the  1995  NRC  report  (commonly  referred  to  as  the  “Press”  Report)  is  the  need  for  some 
form  of  "comprehensive"  and  "coherent"  coordination  of  federally  financed  research.  (NAS,  1995)  The  NSB 
concluded  that  such  comprehensive  coordination  of  federally  financed  research  “could  assist  the  President  and 
Congress  by  providing  a  valuable  addition  to  and  improvement  over  the  processes  presently  in  place.  However, 
implementation  of  such  a  policy  would  involve  the  difficult  task  of  developing  acceptable  procedures....This  timely 
and  critical  but  highly  controversial  proposal  merits  careful  attention  at  this  time.”  (NSB  1997) 

One  implicit  critique  would  appear  to  be  pulling  in  the  opposite  reaction.  This  would  advocate  a  shift  from 
the  current  basic  budgetary  stance  where  R&D  expenditures  are  set  de  facto  as  a  fixed  proportion  of  the  federal 
budget  to  one  where  the  goal  of  federal  R&D  funds  is  explicitly  to  support  the  conduct  science,  not  to  achieve  some 
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broader  ends.  (Lederman  1991,  p.  1 8)  The  proposal,  certainly  part  of  a  nuanced  and  sophisticated  argument,  would 
suggest  a  less  instrumental  orientation  toward  science  as  a  means  to  address  specific  ends  and  more  of  a  stance  that 
viewed  research  as  being  ultimately  useful  but  worthy  of  support  in  itself. 

ADVANTAGES/DISADVANTAGES  OF  DIFFERENT  METHODOLOGIES 

Aspects  of  the  literature  that  explicitly  weigh  the  benefits  and  drawbacks  of  different  methodological 
approaches  would  include  the  Robinson  critique  of  the  Press  report.  His  argument  against  a  more  coherent  and 
integrated  federal  S&T  budget  is  that: 

•  trade-offs  should  not  be  made  between  different  categories  of  federal  S&T  investment,  but  in  as  much  as 
most  such  research  is  performed  to  advance  the  mission  of  a  government  department  or  agency,  the  tradeoff 
should  be  made  at  the  agency  level  between  S&T  investment  and  other  expenditures  of  the  agency 
conducting  the  S&T; 

•  the  Press  report  underestimates  the  value  of  the  mechanisms  already  in  place  that  develop  coherence  in 
S&T  activities  in  specific  areas  and  especially  the  role  of  the  NSTC; 

•  where  cross-cuts  need  to  be  made  for  coordination  purposes,  NSTC  can  consider  trade-offs  between 
compatible  activities  and  not  need  to  engage  in  the  “overly  comprehensive  process”  proposed  by  the  Press 
panel.  (Robinson,  1997) 

This  all  having  been  said,  the  final  point  (although  whether  intended  as  a  policy  prescription  or  a  reductio  ad 
absurdum  argument  is  unclear)  is  that  to  truly  achieve  the  balance  of  S&T  resources  in  one  area  with  those  in 
another  would  require  the  creation  of  one  department  in  agency  in  which  all  the  activities  are  being  carried  out.  (pp. 
218-220) 

A  consultant  report  to  the  Carnegie  Commission  (Shapley,  1992)  notes  that  the  budget  process  “cannot  be 
expected  to  provide  a  method  or  even  an  analytic  framework  for  setting  the  major  priorities  [on  R&D]  in  the 
budget,”  because  the  goals  of  federal  programs  are  simply  too  diverse  and  lack  common  criteria  for  setting  priorities 
among  the  nation’s  goals  (p.  40).  As  a  bottom  line,  after  weighing  various  criticisms  and  suggestions.  Shapely  finds 
the  treatment  of  R&D  has  both  strengths  and  weaknesses  as  played  out  in  the  current  form  of  the  budget  process.  Its 
strengths  include  a)  wide  recognition  of  the  need  for  and  importance  of  R&D;  b)  the  acceptance  of  a  major  federal 
role  in  support  of  R&D;  and  c)  the  fact  that  [writing  in  1991]  “no  one  is  really  against  it.” 

However,  weaknesses  exist  as  well.  R&D  budget  submissions  tend  to  be  treated  on  a  “level  of  effort”  basis 
in  as  much  as  it  is  too  difficult  to  budget  by  individual  projects.  These  are  “very  hard  to  defend”  and  mask 
indivisibilities  and  rigidities  that  might  cause  cuts  to  have  a  more  than  proportionate  effect  and  so,  in  effect,  change 
priority  decisions.  There  is  also  an  inherent  weakness  in  many  justifications  offered  in  support  of  R&D  budgets. 
Precisely  because  of  difficulties  in  making  causal  arguments  and  tracking  benefits  directly  back  to  research 
outcomes,  it  is  difficult  to  persuade  that  a  given  investment  in  R&D  should  occur  this  year  rather  than  next.  And 
finally,  there  is  a  “vulnerability  that  arises  from  the  “expanding  universe”  nature  of  R&D  budgets”  i.e.,  “the  constant 
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expansion  of  areas  of  important  scientific  research  and  opportunities  for  technological  advances  conspires  with  other 
factors  to  make  increases  in  R&D  budgets  necessaiy  year  after  year.”  (pp.  62-67) 

Another  view  suggests  the  present  system  for  budgeting  has  a  subtle  affect  on  priority  allocation  among 
disciplines.  In  the  case  of  several  disciplines,  their  fortunes  are  inordinately  tied  to  the  missions  of  particular 
agencies.  These  disciplines  are  heavily  affected  by  changes  in  agency  programs  leading  to  severe  post  facto  cuts  in 
the  funding  to  these  fields.  In  this  view,  there  is  need  for  explicit  determination  of  what  levels  of  funding  would  be 
appropriate  on  a  field  by  field  basis  with  the  larger  national  interest  in  view.  Principal  policymaking  bodies  need  to 
make  adjustments  to  the  funding  portfolio  when  there  appears  to  be  a  serious  shortfall  in  desirable  investment. 
(Merrill  and  McGeary,  1999) 

EVIDENCE  OF  USE  OF  METHODOLOGIES 

As  noted  above,  most  critiques  are  concerned  witii  higher  aggregate  concepts  and  are  less  likely  to  deal  with 
actual  methods  for  assigning  priority  or  coordinating  budgets.  Instances  in  the  literature  indicating  the  extent  to 
which  different  methods  are  used  have  already  been  presented  above  in  the  discussion  of  methods  and  models 
employed  in  the  research  portfolio  managing  agencies. 

SUGGESTED  APPROACHES  TO  IMPROVING  PROCESS 

The  literature  presents  many  alternatives  to  the  present  processes  of  priority  setting  and  budgetary 
coordination.  The  suggestions  tend  to  fall  into  one  of  three  areas:  alternative  weightings  or  other  means  for 
deriving  priorities  for  federal  research  support  fi-om  larger  national  goals;  suggestions  for  alternative  mechanisms 
for  budget  coordination  and  ensuring  allocation  of  funds  in  accord  with  priorities  within  existing  institutions;  and 
elements  of  a  design  leading  to  changes  in  those  institutional  structures  themselves. 

Priorities  for  Allocation 

In  the  first  area,  there  are  calls  to  make  more  clear  the  raison  d’etre  for  federal  R&D  support.  Some  themes 
recur  and  amount  to  suggestions  for  specific  priority  choices.  One  study,  for  example,  advocates  generating  a 
fonnal  annual  comprehensive  Federal  Sustainable  S&T  budget.  The  budget  would  provide  a  ranking  of  research 
areas  in  relation  to  importance  for  sustainable  development  as  part  of  a  coordinated  cross-agency  and  cross-branch 
allocation  effort.  (NAS,  1995) 

Sustainability  is  not  the  only  issue  to  have  been  elevated  to  a  prominent  place  in  S&T  decisionmaking  at  the 
national  level.  More  broadly,  there  have  been  calls  to  link  the  allocation  of  research  funds  to  the  quest  for 
addressing  societal  goals.  (Carnegie,  1992)  The  comprehensive  OTA  study  on  alternatives  points  out  that  whatever 
criteria  chosen  should  be  made  explicit  to  ease  actual  processes  of  selection  and  allocation.  (OTA,  1991)  Further, 
there  are  no  true  means  for  looking  at  the  federal  research  portfolio  in  its  totality  to  assess  progress  toward  goals  and 
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fliat  the  selection  criteria  most  commonly  s^plied,  scientific  merit  and  mission  relevance,  “are  in  practice  coarse 
filters  ”  (p.  1 1)  This  stu(fy  also  provides  an  adapted  list  of  more  fully  specified  evaluation  for  selecting  among 
competing  initiatives: 

Scientific  Merit: 

1.  Scientific  objective  and  significance 

2.  Breadfii  of  interest 

3.  Potential  for  new  discoveries  and  understanding 

4.  Uniqueness 

Si2sialB.e.psfii§: 

1 .  Contribution  to  scientific  awareness  or  improvement  of  flie  human  condition 

2.  Contribution  to  international  imderstanding 

3.  Contribution  to  national  pride  and  prestige 

Programmatic  Concerns: 

1 .  Feasibility  and  readiness 

2.  Scientific  logistics  and  infiastructure 

3.  Community  commitment  and  readiness 

4.  Institutional  implications 

5.  International  involvement 

6.  Cost  of  proposed  initiative  (Dutton  and  Crowe,  1988  as  in  OTA,  1991,  p.  141) 

In  addressing  these  issues,  the  foimdation  document  of  the  Clinton  Administration’s  science  policy  calls  not 
only  for  a  re-shaping  of  that  policy  to  sustain  U.S.  scientific  preeminence  but  also  to  direct  science  toward 
advancing  the  core  elements  of  the  national  interest:  health,  prosperity,  national  security,  environmental 
responsibility,  and  improved  quality  of  life.  (Clinton  and  Gore,  1994)  As  noted  previously,  NSTC  is  to  play  a 
primary  role  in  ensuring  that  this  cross-product  yields  a  research  portfolio  in  conformity  with  these  goals.  Yet,  a 
Council  on  Competitiveness  report  still  underscores  a  need  for  the  U.S.  to  establish  a  national  consensus  on  R&D 
goals.  (Council  on  Competitiveness,  1996) 

In  addition  to  priorities  set  by  issue  area,  there  are  also  calls  to  do  so  by  stage  of  the  research  and  innovation 
process  or  other  criteria.  For  example,  calls  to  use  federal  research  funding  to  strengthen  education  and  Human 
resources  or  to  focus  on  regional  and  institutional  capacity  would  fall  in  this  category.  (OTA,  1991).  Many  of  these 
also  are  echoed  in  Science  in  the  National  Interest.  (See,  e.g.,  pp.  23, 27)  They  are  formalized  in  such  programs  as 
SBIR,  EPSCOR,  and  less  formally  by  earmarking  or  other  sectoral  allocations.  Similarly,  there  are  also  suggestions 
to  shift  the  focus  of  funding  in  the  federal  R&D  portfolio  dramatically  in  direction  of  support  to  basic  research. 
(CRS,  1995)  But,  on  the  other  hand,  the  Council  on  Competitiveness  states  that  parsing  the  federal  R&D  budget  by 
the  old  definitions  of  basic  and  applied  research  has  proved  politically  ineffective.  It  calls  on  the  U.S.  to  “adopt  a 
new  and  more  up-to-date  vocabulaiy,  one  that  accounts  for  changing  calculations  of  R&D  risk  and  relevance  over 
short-,  mediiun,  and  long-term  horizons.”  (Council  on  Competitiveness,  1996) 
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Mechanisms  for  Allocation 

The  second  major  group  of  alternatives  does  not  address  itself  so  much  to  the  priorities  for  allocation  as  to 
the  mechanisms  by  which  allocations  should  occur.  The  common  characteristic  of  this  group  of  suggestions  is  fliat 
they  would  preserve  existing  institutions  for  federal  research  portfolio  management.  However,  the  means  used  by 

these  institutions  would  pursue  their  efforts  would  change. 

As  noted  above,  alternative  models  for  allocation  already  exist  at  different  agencies.  Peer  review  is  the 
standby  for  allocation  and  selection  at  both  NIH  and  NSF.  The  strong  program  manager  model  is  the  hallmark  of 
DARPA.  NASA’s  formal  apparatus  exists  as  a  mixed  case  (as  do  they  all,  to  one  degree  or  another).  Models 
already  in  existence  in  federal  agencies  may  be  applied  to  other  federal  agencies  in  furtherance  of  particular  goals  or 
program  elements.  The  recent  report  by  the  House  Science  Committee  on  science  policy  suggests,  “other  federal 
agencies  should  consider  increasing  the  use  of  this  method  [peer-reviewed  research  grants]  of  supporting  and 
encouraging  [investigator-driven,  non-commercial]  research.”  (U.S.  Congress,  1998;  p.  16) 

Several  suggestions  have  been  made  for  fundamental  changes  in  allocation  methods.  Some  entail  crafting 
methodologies  based  upon  different  theoretical  bases  than  ones  now  employed.  As  an  example,  one  proposal  would 
utilize  an  options  approach  to  R&D  funding  where  the  portfolio  is  constantly  updated  and  balanced  for  risk  and 
takes  advantage  of  increased  information  availability  over  time.  It  is  not  clear  that  such  an  approach  is  not  already 
employed  informally  (e.g.,  by  DARPA)  but  this  suggestion  would  formalize  the  approach  and  make  it  a  primary 
allocation  mechanism  across  federal  science-funding  agencies.  (Vonortas,  1999) 

Identifying  Areas  of  Special  Interest 

Many  contributions  to  the  literature  view  the  system  as  it  currently  operates  as  largely  successful  for  the  bulk 
of  research  needs  and  requiring  no  fundamental  change.  However,  within  a  pluralistic,  multi-agency  budgeting 
approach,  areas  requiring  special  attention  can  be  identified.  In  the  formulation  by  the  Presidents  of  the  National 
Academies,  these  latter  would  include  initiatives  contributing  to  the  science  base,  especially  in  modifying  its 
direction  of  development  along  several  axes;  S&T  initiatives  tied  to  major  presidential  or  congressional  objectives; 
and  major  (for  which  read  “big  science”)  S&T  initiatives  slated  for  rapid  growth  or  deceleration.  These  last  require 
extra  attention  owing  to  their  large  potential  for  spillover  effects  on  other  agencies.  (NAS/NAE/IOM,  1988) 

The  question  of  how  the  sorting  of  priorities  and  subsequent  aUocation  is  actually  to  occur  is  frequently  left 
unstated.  A  background  paper  prepared  for  flie  National  Academy  of  Sciences  synopsizes  several  studies.  Going 
back  to  tire  vision  of  Vannevar  Bush,  “criteria  for  support  were  inherently  left  to  internal  sorting  within  the 
administrative  structure  he  envisioned....Most  other  reports  have  provided  process  suggestions  for  balancing 
priorities,  but  have  implicitly  left  the  criteria  to  be  used  to  those  engaged  in  the  process.”  (Cook-Degan,  p.  23)  The 
report  notes  that  both  the  Carnegie  Commission  and  OTA  [in  the  studies  cited  above]  suggested  House  hearings,  for 
example.  But  it  goes  on  to  state  that  “the  most  promising  leads  for  crafting  criteria  of  use[in  crafting  criteria  for 
priority]  may  not  come  from  reports  that  have  not  [sic]  been  policy  reports  covering  all  R&D... but  instead  from 
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approaches  taken  within  fields.”  He  cites  the  example  of  the  1972  NRC  report  Physics  in  Perspective  which 
discusses  criteria  internal  to  fields  wifliin  physics  and  another  set  external  to  science  that  can  help  assess  tiheir 
relative  priority.  “The  committee  came  up  with  a  set  of  18  questions,  and  then  used  the  criteria  on  subfields  within 
physics  and  found  a  sinprising  degree  of  consensus.” 

Finally,  in  this  connection  several  reports  point  up,  on  the  one  hand,  the  paucity  of  data  gathering  and,  on  the 
other,  the  necessity  for  establishing  a  database  on  the  entirety  of  the  federal  R&D  budget  for  the  purposes  of  priority 
setting,  coordination,  and  evaluation.  (OTA,  1991;  Shapley,  1992) 

Modifying  the  Priority-Setting  Institutions 

The  last  category  of  suggestion  and  alternatives  to  be  found  in  the  literature  are  those  which  address 
themselves  to  and  advocate  modification  of  the  institutions  of  federal  research  portfolio  management  themselves. 

Perhaps  the  most  fi'equent  observation  in  the  literature  we  reviewed  was  that  the  federal  R&D  portfolio  is 
only  a  post  facto  accounting  concept.  It  is,  by  default,  the  aggregation  of  individual  mission  agency  portfolios  but  is 
in  no  sense  managed  as  a  unified  portfolio.  Several  studies  advocate  a  greater  role  for  NSTC,  OSTP,  and/or  0MB  in 
setting  portfolio  guidelines  at  a  higher  level  dian  that  of  the  funding  agencies  as  well  as  actively  monitoring 
fulfillment,  (see,  e.g.  Carnegie,  1992)  Formally  speaking  these  are  institutions  already  in  existence.  Yet,  as  a 
practical  matter  this  would  constitute  a  fundamental  redrafting  of  the  role  of  these  agencies  and  the  nature  of  their 
interactions  with  the  rest  of  the  federal  research  portfolio  management  structure. 

A  number  of  reports  have  called  for  improved  scientific  coordination  across  the  agencies.  Willis  Shapley 
makes  three  recommendations:  1)  establish  a  comprehensive,  comparable  data  base  on  R&D  budgets;  2)  create  a 
detailed  “directory”  report  that  could  provide  comprehensive  information  on  what  agencies  are  engaged  in  what 
kinds  of  R&D;  and  3)  prepare  a  report  on  “R&D  in  the  Budget”  each  year.  (Shapley,  1992) 

Alongside  suggestions  for  different  goals  stands  the  recommendation  that  a  new  institutional  structure  for 
setting  such  goals  be  created.  One  paper  advocates  establishing  a  non-govemmental  National  Forum  on  S&T  to  set 
the  goals  for  the  entire  establishment.  Upon  its  creation,  such  a  body  would  first  identify  the  de  facto  goals  that  are 
presently  operative  based  on  past  budgets.  It  would  then  develop  the  means  for  defining  goals  for  the  system  of 
federally  fimded  research  within  the  context  ofbroader  societal  or  governmental  goals.  (Carnegie,  1992)  Another 
suggestion  along  these  lines  would  be  for  such  a  body  to  define  what  die  essential  elements  in  the  federal  R&D 
portfolio  must  be  and  suggest  ways  in  which  the  portfolio’s  composition  may  be  more  readily  adapted  to  changing 
national  circumstances.  (Hill,  1995) 

There  are  also  suggestions  whose  effect  would  be  to  provide  a  variation  on  the  Vannevar  Bush  vision  of  a 
single  National  Research  Foundation.  Given  the  greater  mission  orientation  and  institutionalization  of  federal 
decisionmaking  that  characterizes  the  present  compared  with  Bush’s  time,  die  proposal  is  less  for  such  an  agency 
than  to  provide  centrality  by  actually  formulating  and  passing  through  Congress  a  Federal  S&T  Budget.  Rather  than 
existing  as  a  post  hoc  accounting  concept  that  is  the  construct  of  a  bottom  up  process,  such  a  budget  would  be 
designed  as  an  instrument  not  so  much  for  allocation  as  for  the  setting  of  allocative  priorities  in  a  top  down  manner. 
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(NAS,  1995)  This  proposal,  for  good  or  ill,  would  make  more  visible  the  funding  activities  supporting  federal 
efforts  in  supporting  S&T  as  well  as  to  make  clearer  the  tradeoffs  required  to  balance  the  public’s  interest  in  this 
area  of  policy.  In  a  further  discussion  of  this  idea.  Press  (1997)  makes  the  points  that  the  FS&T  budget  would  be 
focused  on  more  narrowly  defined  activities  than  are  presently  referred  to  as  “R&D”  so  that  the  principal  recipients 
of  federal  funding  would  shift  fi-om  industrial  contractors  to  federal  laboratories  and  universities.  It  would  work  by 
forcing  trade-offs  by  “transfer  of  funds  fi-om  poorly  evaluated  or  obsolete  programs  to  those  of  higher  quality  or 
those  more  responsive  to  social  and  economic  exigencies...The  FS&T  budget  is  an  instrument  that  looks  at  the  real 
R&D  pool,  moves  funds  around  within  it,  and  keeps  a  running  score  of  what’s  happening.”  (p.  175)  The  mam 
purpose  is  to  protect  the  worthy  programs  from  being  “picked  off  [by]  mindless  decisions.” 

Arguments  for  Maintaining  the  Current  Pluralistic  System 

Although  a  number  of  reports  and  papers  recommend  alternatives  to  the  current  system  of  budgeting  and 
priority  setting,  not  all  of  them  call  for  change.  Just  as  Willis  Shapley  argued  in  “The  Budget  Process  and  R&D”, 
cited  above,  that  priorities  must  be  set  within  the  political  process,  Christopher  Hill,  David  Robinson  and  Albert 
Teich  make  similar  arguments  in  other  publications.  Hill  notes  that  “federal  R&D  spending  levels  are  determined  in 
a  “profoundly  political  process,  representing  a  balance  among  competing  interests,  not  only  with  respect  to  R&D, 
but  also  with  respect  to  all  other  claims  on  the  federal  treasury  (Hill,  1995,  p.  5 ),  and  he  asserts  that  this  process 
serves  science  fairly  well.  Even  the  National  Academy  of  Sciences,  in  a  1988  report,  notes  that  the  pluralistic 
approach  to  budgeting  has  been  a  strength  of  the  U.S.  system.  (NAS  reports  in  the  1990s  take  a  different  view.) 

In  a  paper  written  in  response  to  the  NAS  report  Allocating  Federal  Funds  for  Science  and  Technology, 
Robinson  says  “. .  .trade-offs  should  not  be  made  between  different  categories  of  FS&T  investment,  but  between 
S&T  investment  and  other  expenditures  of  the  agency  conducting  S&T ...”  just  as  it  is  now.  Moreover,  Robmson 
argues  that  the  Press  panel  underestimates  the  value  of  the  current  mechanisms  to  develop  a  coherent  S&T  program. 
The  NSTC  “crosscuts”  have  improved  the  efficiency  of  ftie  process.  Similarly,  Teich  notes  that  priority-setting  is  an 
integral  part  of  the  regular  budget  process  and  “not  something  done  internally  by  the  scientific  community  and  then 
handed  over  to  government. . .”  Moreover,  Teich  notes  fiiat  which  groups  should  speak  for  science  is  not  at  all  clear 

Advocates  of  priority-setting  have  suggested  that  it  is  important  for  the  scientific  community  to  present  a 
unified  appearance  to  the  outside  world.  But  who  can  speak  authoritatively  on  behalf  of  toe  scientific 
community?  A  committee  ofthe  National  Academy  of  Sciences?  The  scientific  associations?  (If  so, 
which?)  OSTP?  This  is  not  a  trivial  matter.  Whichever  body  conducts  the  priority-setting  exercise  has  to  be 
broadly  accepted  in  all  fields  of  science  and  legitimate  and  powerful  enough  to  stand  behind  what  are  bound 

to  be  unpopular  choices(Teich,  1991,  p.  43).  r-  j  ar  j 

These  and  several  others,  including  Hill  (1995)  suggest  that  the  current  process  of  trade-offs  and  political 

decisionmaking,  influenced  by  advocates  of  science,  actually  works  fairly  well  and  meets  the  needs  of  science  for 
adequate  funding. 
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5.  OBSERVATIONS  AND  SUGGESTIONS 


In  the  preceding  pages  this  report  has  presented  an  analytical  review  of  the  literature  pertammg  to  several 
issue  areas  touching  upon  federal  R&D  budget  priority  setting  and  coordination  across  fields  of  science.  To  the 
extent  fliat  the  literature  spoke  widi  a  clear  voice,  we  have  sought  to  present  that  voice.  When  tiie  voice  was  more 
subdued  or  oblique,  we  attempted  to  present  the  main  lines  of  discourse  in  a  manner  that  would  illuminate  the  issues 
of  interest  to  NSB.  Synopses  of  the  literature  provided  in  this  document  had  the  intention  of  capturing  the  essence 
of  what  these  voices  were  conveying.  Where  the  voice  was  silent,  we  have  noted  its  absence. 

In  what  follows,  we  offer  several  insights  drawn  by  the  research  team  during  the  course  of  its  endeavors. 

The  spirit  which  informs  this  effort  is  less  to  provide  a  self-contained  collation  and  synthesis  of  the  literature  than  to 
offer  propositions  for  discussion  and  further  refinement  through  Committee  interactions  and  between  the  Committee 

and  the  NSB  staff. 

Overall  Assessment  of  the  Literature 

The  literature  is  skewed  toward  the  prescriptive  rather  than  the  descriptive  and  tends  to  take  a  broad  view  rather  than 
examine  operations  at  the  agency  level.  Thus,  few  fonnal  models  of  current  priority  setting,  allocation,  and 

coordination  have  been  identified. 

The  literature  as  it  stands  falls  into  three  categories: 

•  A  robust  literature  offering  advice  to  government  on  how  best  to  set  goals  and  allocate  funds, 

both  as  a  national  endeavor  and  across  govermnental  agencies; 

•  A  smaller  but  growing  base  of  procedural  publications  describing  how  the  process  of  research 
and  development  (R&D)  allocation  should  be  done  within  an  agency  or  a  discipline; 

•  A  few  examples  of  how  the  process  actually  takes  place  within  the  government. 

There  is  little  in  the  literature  describing  efforts  at  coordination  other  than  those  of  the  NSTC,  either  among 
scientific  programs  across  agencies,  or,  efforts  to  coordinate  programmatic  activities  among  different  fields 

of  science. 

More  attention  is  paid  to  goal  setting  than  coordination.  Recommendations  fall  into  three  broad  categones: 

•  A  shift  to  a  national-goals  approach,  tying  the  priority  setting  process  to  national  goals, 

•  A  scientific-goals  approach  advocating  cross-cut  assessments  of  existing  spendmg  m  areas  of 
science  and  realignment  of  budgets,  if  needed,  to  better  meet  the  needs  of  scientific 
advancement;  and 

•  Fine-tuning  of  the  existing  complex,  political  process. 

•  There  is  a  richer  discussion  on  goals  and  priority  setting  within  the  executive  branch  than  within 
Congress.  Qualitative  discussions  of  how,  or  even  whether.  Congress  decides  between  funding  options 
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for  different  areas  of  science,  different  federal  R&D  programs,  or  different  research  project  areas  are 
comparatively  rare. 

•  Agencies  differ  in  setting  priorities  for  science  based  on  whether  they  have  a  scientific  or  mission 
orientation.  Most  agencies  now  use  some  combination  of  an  outreach  or  advisory  -approach  to  priority 
setting  by  gathering  in  views  from  various  stakeholders,  combined  with  a  strategic  planning  approach, 
setting  strategic  goals  and  seeking  to  implement  them.  How  these  two  functions  interact  within  the 
agencies  to  affect  priorities  is  not  detailed  in  the  literature. 

Detailed  Consideration  of  Key  Gaps  in  the  Literature 

•  There  is  relatively  little  detail  about  the  actual  operations  of  the  NSTC  in  coordinating  federal  R&D 
policy,  planning,  and  budgets.  The  literature  cites  the  role  of  the  NSTC  in  coordinating  larger 
initiatives  and  crosscuts,  although  the  actual  importance  of  the  NSTC  committees  versus  decisions  by 
die  0MB  budget  examiners  or  Assistant  Directors  is  not  detailed.  Likewise  there  is  little  description  of 
the  NSTC  role,  if  any,  in  determining  funding  in  agency  core  R&D  programs  not  connected  to  a  larger 
budget  priority  or  “crosscut.” 

•  There  is  little  to  no  description  of  the  process  that  takes  place  within  the  Executive  Branch  in  the  1 1 
months  leading  up  to  the  release  and  explanation  of  the  President’s  budget.  The  development  of 
agency  budgets  can  be  complicated,  especially  in  agencies  with  both  R&D  and  operational  programs 
(DOE,  NOAA,  USDA,  etc.).  The  struggle  within  agencies  for  resources  in  the  period  prior  to  the 
submission  of  the  proposed  budget  to  0MB  in  September  is  nowhere  in  the  literature.  Yet  this  process 
sets  the  terms  of  negotiations  between  0MB  and  the  agencies  for  the  rest  of  the  year. 

•  Likewise,  the  give  and  take  between  0MB  and  the  agencies  prior  to  the  agencies  being  “locked  ouf  ’  of 
the  budget  in  December  is  not  described.  There  is  no  mention  of  how  the  individual  divisions  of  0MB 
make  decisions,  set  priorities,  or  allocate  funding.  To  the  point  of  this  paper,  there  is  no  mention  made 
how,  or  even  whether,  there  are  any  attempts  made  at  0MB  to  coordinate  R&D  poliQ^,  planning  or 
budgets  between  agencies  and  departments. 

•  The  readjustment  of  the  budget  that  occurs  after  the  agency  “lock  out”  in  December,  and  when  final 
Presidential  priorities  are  set,  prior  to  the  budget  being  final  in  January,  are  not  described  in  the 
literature.  Yet  many  recent  significant  R&D  budget  and  policy  decisions  have  been  made  in  this  final 
administration  budget  deliberation. 

•  Likewise,  there  is  little  description  of  the  various  ways  in  which  Congressional  committees  influence 
tile  direction  and  conduct  of  federal  R&D  through  a  number  of  informal  means.  The  language 
contained  in  appropriations  bill  reports  carries  the  force  of  law  and  it  is  in  those  reports  that  many 
qualitative  decisions  are  made  that  influence  an  agency’s  priorities.  Even  telephone  communications 
between  key  appropriations  staffers  and  agency  administrators  have  resulted  in  significant  shifts  of 
funding  or  R&D  program  direction.  Location-specific  earmarks  are  only  one  form  of  this 
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Congressional  communication.  The  initiation  of  a  major  federal  R&D  effort  on  breast  cancer  at  DOD 
was  the  result  of  Congressional  intervention  in  the  priority  and  funding  process.  Most  of  these 
decisions  are  made  without  reference  to  what  deliberative  process  that  was  used  to  make  them  and 
scholarly  documentation  is  rare. 

•  There  is  no  published  explanation  for  the  current  balance  or  trends  among  research  accounts  (shown 
most  recently  in  the  paper  by  Merrill  and  McGeaiy).  Major  decisions  are  being  made  outside  of  any 
formally  described  R&D  policy,  planning,  or  funding  process.  The  chart  in  Merill  and  McGeaiy  on 
funding  trends  by  discipline  is  not  accompanied  by  any  explanation  of  how  Congress  or  the  agencies 
involved  arrived  at  these  decisions  or  whether  this  was  even  the  result  of  any  premeditated  process. 

•  Coordinate  with  the  above  point,  as  rare  as  detailed  descriptions  of  the  formal  priority  setting 
mechanisms  of  federal  R&D  managing  agencies  are,  rarer  still  are  documents  elaborating  on  either  the 
details  of  these  procedures  in  practice  or  the  degree  to  which  that  practice  actually  corresponds  to,  or 
deviates  from,  the  formal  procedures. 


Data  Issues 

•  Data  on  the  contents  of  the  federal  R&D  portfolio  contain  either  highly  aggregated  budget  infonnation 
or  disaggregated  project  descriptions.  There  is  considerable  difficulty  in  finding  common  bases  for 
combining  data  collected  by  different  agencies. 

•  A  considerable  problem  when  concerning  existing  databases  is  mixed  definitions.  The  terms  “R&D” 
and  “S&T’  are  often  used  as  synonyms.  Not  only  do  these  terms  cany  different  meanings  in  the 
defense  and  civilian  agencies,  S&T  carries  multiple  meanings.  Lack  of  awareness  of  this  continually 
impedes  efforts  to  improve  planning,  management,  and  coordination  of  federal  science  activities. 

Uniqueness  of  the  R&D  Planning  Problem 

The  coordination  of  federal  R&D  functions  has  been  the  subject  of  considerable  attention  and  deliberation. 
These  efforts  began  with  the  Science  Policy  Act  of  1 976,  have  found  new  form  with  the  development  of  instruments 
such  as  die  NSTC,  and  certainly  continue  on  through  to  the  NSB’s  own  inquiry  today.  It  might  be  worth  noting 
explicitly  just  how  remaricable  both  fliis  effort  and  the  area  of  federal  activity  it  seeks  to  address  truly  are.  One  is 
hard  pressed  to  name  other  vital  federal  government  functions  in  the  discretionary  part  of  the  budget  that  cross  so 
many  Executive  agency  and  Congressional  committee  lines.  It  is  hard  to  think  of  any  parallel  federal  activity  that 
has  such  importance  and  such  a  broad  involvement  of  so  many  agencies.  To  be  sure,  fliere  are  programs  seeking  to 
provide  public  assistance,  income  support,  and  transfer  payments  spread  about  the  federal  structure,  but  those  are  for 
die  most  part  mandatory  programs.  Other  vital  government  functions  are  contained  in  single  agencies,  education 
finally  was  given  its  own  department,  as  were  health,  housing,  veteran  affairs,  etc.  There  may  be  parallels  in  such 
fields  as  natural  resource  planning  or  issues  related  to  the  use  of  nuclear  power,  but  they  involve  only  a  few 
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agencies.  And  if  this  were  not  enough,  the  R&D  planning  problem  needs  not  only  to  be  funded,  managed,  and 
administered  across  the  government,  it  needs  to  be  applied  across  quite  disparate  fields  of  science. 

The  discussion  of  coordination  issues  presented  in  this  literature  review  points  up  the  unique  nature  of  the 
R&D  coordination  and  priority  setting  problem.  What  does  this  imply  for  the  NSB’s  explorations  and  deliberations? 
In  essence,  if  it  is  true  that  there  are  no  other  federal  activities  that  have  to  coordinate  their  functions  across  so  many 
agencies,  any  coordination  effort  will  to  a  large  extent  need  to  make  its  own  way  without  other  successful  models  of 
coordination  to  borrow  from.  This  certainly  suggests  the  value  of  beginning  with  first  principles  in  considering  what 
may  be  desired  in  this  area  and  what  efforts  may  be  needed  to  bring  it  about. 


Alternative  Approaches  to  Management 

The  alternative  approaches  for  managing  the  federal  research  enterprise  we  have  detected  in  the  literature  fall 
into  three  areas:  1)  alternative  readings  of  what  are  the  appropriate  goals  federal  research  support  should  seek  to 
fulfill;  2)  suggestions  for  alternative  mechanisms  for  allocation  of  funds  within  the  existing  institutions  for  managing 
the  federal  research  portfolio;  and  3)  elements  for  a  design  leading  in  whole  or  in  part  to  changes  in  those 
institutionaljtructures  themselves.  The  two  consistent  threads,  present  in  varying  degrees,  are  a  desire  to  establish 
priorities  and  to  do  so  in  a  coordinated  fashion. 

This  is  understandable.  A  frequent  theme  of  the  literature  is  that  the  federal  R&D  portfolio  is  only  a  post 
facto  accounting  concept.  It  is,  by  default,  the  aggregation  of  individual  mission  agency  portfolios  but  is  in  no  sense 
managed  ex  ante  as  a  unified  portfolio.  Several  studies  advocate  a  greater  role  for  the  NSTC,  OSTP,  and/or  OMB  in 
setting  portfolio  guidelines  at  a  higher  level  than  that  of  the  funding  agencies  as  well  as  actively  monitoring 
fulfillment.  A  necessary  first  step  to  effective  prioritization,  in  other  words,  is  to  achieve  coordination.  Yet,  such  a 
quest  may  prove  to  be  chimerical. 

As  we  have  suggested,  “coordination”  may  refer  to  hierarchical  synchronization  as  a  means  to  ensure  that 
larger  goals  are  being  met  through  the  smaller-scale  decisions  over  priority  and  allocation.  Alternatively,  the  term 
refers  to  cross-cutting  coordination  to  ascertain  that  the  actions  of  individual  agencies  do  not  excessively  duplicate 
or  pre-empt  those  of  other  agencies.  The  two  themes  are  highly  intertwined. 

Is  Coordination  Really  Deficient? 

A  persistent  assumption  in  the  literature  is  that  greater  coordination  is  desirable  and  can  be  attained  by 
setting  high  level  goals  and  then  proceeding  to  lower  levels  of  decisionmaking  authority.  This  seems  problematic  on 
two  counts.  First,  decision  making  in  this  area  is  embedded  in  existing  institutions  and  political  processes.  Setting 
high-level  goals  and  then  rigorously  enforcing  them  as  the  means  for  crafting  priorities  and  making  allocations  on 
lower  levels  would,  in  effect,  stand  the  present  system  on  its  head. 

Second,  such  an  approach  may  not  accord  with  the  evolving  pattern  for  the  commerce  of  ideas  and 
knowledge  management.  Modem  science  is  increasingly  cross-disciplinary,  with  major  discovery  taking  place  at 
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tfae  interstices  of  traditional  disciplinary  categories.  Contributing  to  this  trend  are  increased  globalization  of  effort 
with  geographically  dispersed  working  teams  crossing  geographic  boundaries  and  an  ever  denser  connectivity  of 
information  and  ideas. 

This  is  not  to  suggest  that  prioritization  or  coordination  are  imdesirable  or  that  gaining  a  measure  of  control 
is  impossible.  Rather,  it  is  to  suggest  we  think  about  the  following  questions: 

1 .  What  do  we  mean  by  the  terms  “priority”  and  “coordination”? 

2.  What  do  we  hope  to  achieve  and  how  will  we  improve  the  public’s  lot  through  prioritization  and 
coordination? 

3.  What  alternative  models,  not  necessarily  predicated  upon  traditional  views  of  either  the  science  process 
or  its  effect  on  the  larger  society,  need  we  consider  to  best  develop  means  for  achieving  a  true  ability  to 
set  priority  and  the  level  of  coordination  we  desire? 
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SPECIAL  ISSUES  SECTION  1.  THE  ISSUE  OF  DEFINITIONS:  “R&D”  VERSUS 
“S&T”  AND  GOVERNMENT  DATA  SOURCES 


The  National  Science  Board  has  formally  recognized  the  importance  of  explicitly  addressing  the  definitions  of 
"research"  and  "development"  and  has  stated  that  “considerable  confusion  has  been  created  by  imprecise  and 
sometimes  improper  use  of  the  teirn  R&D.  Die  Board  feels  it  is  important  to  clarify  this  issue.”  (NSB,  1997).  The 
discussion  then  considers  the  nature  of  the  two  distinct  but  interrelated  activities  from  a  substantive  perspective. 
However,  the  problem  alluded  to  by  NSB  is  even  more  general  and  subtle  than  this  document  would  suggest. 

In  particular,  although  the  tenns  “science  and  technology”  (S&T)  and  “research  and  development”  (R&D)  are 
often  used  interchangeably,  ftiey  have  veiy  different  meanings  in  the  context  of  the  federal  government.  Specifically, 
there  is  one  overarching  definition  of  R&D  used  by  all  federal  agencies.  There  are  at  least  two  conflicting  meanings 
of  S&T,  however,  neither  of  which  is  synonymous  with  R&D.  A  lack  of  awareness  of  these  differences  has 
continually  impeded  efforts  to  plan,  manage,  and  coordinate  federal  R&D  activities.  It  has  also  had  an  effect  on  the 
allocation  of  federal  R&D  resources.  In  the  literature  examined  in  this  projea,  numerous  examples  were  found  of 
how  the  different  and  conflicting  meanings  attached  to  these  terms  by  various  parts  of  the  federal  government  have 
also  made  it  difficult  to  set  priorities  and  coordinate  activities  among  federal  agencies. 

WHAT  IS  “RESEARCH  AND  DEVELOPMENT”  WITHIN  THE  FEDERAL  GOVERNMENT? 

R&D  is  a  budget  category  that  is  defined  by  die  Office  of  Management  and  Budget  (OMB)  in  Section  25  of 
Circular  A-1 1.  The  overall  purpose  of  Ais  circular  is  to  provide  detailed  instructions  and  guidance  to  federal  agencies 
regarding  the  preparation  and  submission  of  their  aimual  budget  requests,  as  well  as  their  strategic,  performance,  and 
capital  asset  plans.  The  specific  purpose  of  Section  25  is  to  designate  the  character  of  all  federal  funds  by  noting  if 
they  are  purchasing  an  investment  whose  benefit  wiU  not  be  realized  for  some  time  or  a  non-investment  whose  benefit 
is  more  immediate.  Because  R&D  activities  constitute  the  primary  long-term  investment  of  the  federal  government 
(education  and  training  is  number  two),  R&D  is  separately  tracked  in  the  federal  budget.  As  noted  in  the  left  side  of 
the  chart  below,  the  OMB  definition  of  R&D  is  composed  of  five  parts:  Basic  Research,  Applied  Research, 
Development,  R&D  Equipment,  and  R&D  Facilities.  The  first  three  of  these  categories  are  collectively  referred  to  as 

the  “Conduct  of  R&D.” 

As  noted  in  the  right  side  of  Table  2.1  below,  the  Department  of  Defense  (DOD)  alone  among  the  federal 
agencies  has  refined  the  government-wide  OMB  definition  by  sub-dividing  the  Development  category  to  distinguish 
among  the  many  R&D  activities  that  tend  to  be  unique  to  DOD.  DOD  then  takes  one  of  these  sub-categories  of 
Development  and  groups  it  with  Basic  Research  and  Applied  Research  and  collectively  refers  to  these  three  activities 
as  S&T.  In  so  doing,  DOD  officially  designates  S&T  as  a  sub-set  of  R&D  for  the  militaiy  activities  of  the  federal 

government. 
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Table  SSl.l  Government-Wide  and  DOD  Definitions  of  R&D 


Definition  of  R&D 


Government'Wide 


0MB  Circular  No.  A-1 1  (1998) 


c  Research 


DOD>Unique* 


DOD  Financial  Management  Relation  (Volume  2B»  Chapter  S) 


pplied  Research 


-  - 


Systematic  study  directed  toward  greater 
aiowJedge  or  understanding  of  the 
'undamental  aspects  of  phenomena  and  or 
observable  facts  without  specific  ^plication 
owards  processes  or  products  in  mind. 


fl 


derstanding  necessary  to  determine  the 
neans  by  which  a  recognized  and  specific 
eedmay  be  met. 


ystematic  fqoplication  of  knowledge  towarc 
le  production  of  useful  materials,  devices, 
nd  systems  or  methods,  including  design, 
evelopmcnt,  and  improvement  of 
rototypes  and  new  processes  to  meet 
pedfic  requirements. 


■fj'’  .'‘Jill 


Deludes  all  efforts  that  have  moved  into  the 
evelopment  and  integration  of  hardware  for 
ield  experiments  and  tests. 


Deludes  all  effects  necessary  to  evaluate 
tegrated  technologies  in  as  realistic  an 
lerating  environment  as  possiUe  to  assess 
e  performance  or  cost  reduction  potential 
f  advanced  tedmology. 


Deludes  R&D  efforts  directed  toward 
upport  of  installations  ch*  operations 
ired  for  general  R&D  use.  Included 
Id  be  test  ranges,  military  construction, 
naimenance  support  of  laboratories, 
erations  and  mairnenance  of  test  aircraft 
d  ships,  and  studies  and  analyses  in 
upport  of  R&D  program. 


I 


\im 


EquijHnent 


e  acquisition  of  major  equipment  for  R&D.  Includes 
cpend^le  or  moveable  equipment  (e.g.,  spectrometers, 
icroscopes)  and  ofiPice  furniture  and  equipment  Routine 
irchases  ordinary  office  equipment  or  furniture  and  fixtures 
e  normally  exclud^. 


li 


brts  associated  with  engineering  or 
upport  activities  to  determine  the 
cceptability  of  a  system,  sub^stem,  or 
ponent. 


brts  associated  with  engineering  or 
iqiport  activities  to  determine  the 

ili^  of  a  system,  sub^stem,  or 
mponent. 


ijor  equipmem  dollars  are  mixed  with  the  dollars  for  the 
Conduct  of  R&D"  and  carried  in  the  RDT&E  accounts  (i.e.,  6. 1 
ough  6.7)  listed  above  In  F  Y  1998,  DOD  requested  a  total  of 
8M  for  m^or  R&D  equipment. 


e  construction  and  renabilitation  ot  K&U  tacilities.  Includes  |Mo  separate 
e  acquisition,  design,  and  construction  of,  or  major  rq^airs  or  iefinition 
terations  to  all  f^ysical  facilities  for  use  in  R&D  activities. 

'acilities  include  land,  buildings,  and  fixed  capital  equipment, 
gardless  of  whether  the  facilities  are  to  be  used  by  the 
ovemment  or  by  a  private  organization,  and  regardless  of  where 
itle  to  the  property  m^  rest.  Includes  such  fixed  facilities  as 
eactors,  wind  tunnels,  and  particle  reactors.  Excludes  movable 
t&D  equipment. 


Does  not  pertain  to  the  Corps  of  Engineers. _ _ _ _ _ 

**  Includes  administativc  expenses.  Excludes  routine  product  testing,  quality  control,  mapping,  collection  of  general-purpose  statistics,  experimental  production,  routine 
onitoring  and  evaluation  of  an  operational  program,  and  the  training  of  scientific  and  technical  personnel. 


nnel,  either  in-house  or  contractor  o 


SSSSI 
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WHAT  IS  “SCIENCE  AND  TECHNOLOGY  ”  WITHIN  THE  FEDERAL  GOVERNMENT? 


Parallel  to  the  designation  of  specific  federal  activities  as  R&D  there  is  a  simultaneous  labeling  of  these  and  all 
other  federal  activities  as  to  their  general  purpose  (i.e.,  mission)  or  function.  Listed  in  Table  2.2  below  are  the 
substantive  functions  among  which  all  federal  activities  are  allocated. 


Table  SS1.2  Functional  Designations  of  R&D  Activities 


Functional  Designations  of  R&D  Activities* 

Function 

Code 

Primaiy  Purpose  of  Activity 
(Function) 

Agencies  Funding  R&D  in  Area 

050 

National  Defense 

DOD  (Military),  DOE 

150 

AID.ACDA  . 

250 

General  Science,  Space  and  Technology 

DOE,  NASA,  NSF 

270 

Energy  _ _ _ 

D0E,TVA,NRC 

300 

Natural  Resources  and  Environment 

DOD  (Civilian),  DOI.  USDA  DOC,  DOT,  EPA,  MMC 

350 

USDA 

370 

Conunerce  &  Housing  Credit 

DOC.USPS 

400 

NASA,  DOT 

450 

Community  &  Regional  Development 

DOC,TVA,HUD 

500 

Educational,  Training,  Employment  &  Social  Services 

HHS,  DED,  DOL,  SMITH 

550 

Health 

DOD  (Military),  HHS,  DOL 

570 

HHS 

600 

Income  Security 

DOL.  SSA 

650 

SSA 

700 

Veterans  Benefits  &  Services 

DVA 

750 

Administration  of  Justice 

DOLTREA 

800 

General  Government 

TREA 

*  TheBuc 
activity's  n 
provide  a  c 
two  or  moi 
Used  in  th 

leet  Function  is  a  classification  made  without  regard  to  agency  or  organizational  distinction  that  best  defines  the 
wst  important  purpose  in  terms  of  addressing  one  of  17  national  needs  These  national  n^s  ^  broad^^  u^d  to 
»heientand  comprehensive  basis  for  analyzing  and  understanding  ihe  federal  budget.  A  function  may  be  divi^  into 
re  subfunctions,  depending  upon  the  complexity  of  the  national  need  addressed  by  the  function.  {A  Glossary  of  Term 
e  Federal  Budget  Process  ,  GAO  1993)  _ _ _ 

As  is  clear  in  the  chart  above,  R&D  activities  are  found  in  every  functional  category  in  the  federal  budget.  Of 
particular  interest  are  the  R&D  activities  in  Function  250  -  General  Science,  Space  and  Technology.  This  functional 
category  encompasses  the  National  Science  Foundation  (NSF)  and  a  large  portion  of  the  Department  of  Energy 
(DOE),  in  addition  to  the  National  Aeronautics  and  Space  Administration  (NASA).  As  a  result,  all  of  the  activities  in 
these  functional  categories,  most  especially  those  of  NSF  and  much  of  DOE,  are  officially  labeled  as  S&T  activities. 
Only  apart  of  the  activities  in  each  of  these  functional  categories,  however,  are  categorized  as  R&D.  Consequently, 
for  these  agencies,  R&D  is  a  sub-set  of  S&T. 
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Table  SS1.3  R&D  vs.  S&T  by  Agency  and  Function 


R&D  vs.  S&T  by  Agency  and  Function 

(Atl  dollars  are  Budget  Authority  in  Millions  for  FY 1998) 

Agency* 

Main 

Budget 

Function 

Code 

National  Need/Goal 
(Sub-Function  Category) 

Total 

Dollars 

in 

Category 

R&D 

Dollars 

in 

Category 

S&T 

Elollars 

in 

Category 

DOD 

Wfater  resourws** 

? 

HHS 

■ 

Social  services 

Health  care  services 

Health  research  &  training 

Consumer  &  occupational  health  &  safety 

Medicare 

13,413 

114,262 

14,310 

926 

284,710 

25 

197 

13,387 

159 

50 

■ 

NASA 

n^n 

1,326 

7 

■ 

050 

^*""27cr 

270 

Atomic  energy  defense  activities 

Energy  supply 

Energy  conservation  . 

3.012 

881 

356 

? 

? 

NSF 

^^582^ 

USDA 

300 

350 

Conservation  &  land  management 

Agricultural  research  &  service 

5,986 

3,005 

191 

1,356 

7 

7 

300 

370 

450 

Other  natural  resources 

Other  advancement  &  regulation  of  commerce 

Area  &  regional  development 

574 

504 

1 

■ 

■ 

300 

400 

400 

400 

400 

Pollution  control  &  abatement 

Ground  transportation 

Air  transportation 

Water  transportation 

Other  transportation 

131 

28,683 

9,161 

3,710 

185 

4 

387 

242 

16 

9 

■ 

03/ 

7 

■ 

300 

300 

300 

300 

Water  resources 

Conservation  &  land  management 

Recreational  resources 

Other  natural  resources 

994 

1,574 

3,919 

767 

10 

36 

171 

403 

7 

7 

7 

7 

DVA 

■EH 

Hospital  &  medical  care  for  veterans 

Other  veterans  benefits  &  services 

18,541 

942 

604 

4 

7 

7 

500 

500 

500 

500 

tiementary,  seconaary  &  vocational  education 

Higher  education 

Research  &  general  education  aids 

Social  services 

1b,2db 

13,074 

772 

2.592 

■ 

■ 

International  development  &  humanitarian  assistance 
International  security  assistance 

3,690 

na 

mM 

7 

7 

Research  &  general  education  aids 

490 

146 

? 

DUC - 

500 

550 

600 

1  raining  &  employment 

Consumer  &  occupational  health  &  safety 

General  retirement  &  disability  insurance  (non-SS) 

6,3/0 

539 

1,246 

—  '■^>5' 
5 
1 

■ 

d5j 

75o 

750 

Federal  law  enforcement 

Criminal  iustice  assistance 

3 

56 

7 

7 

mamm 

375 

56 

7 

NRC 

276 

468 

51 

7 

HUD 

450 

Community  development 

37 

7 

TVA 

“575“ 

450 

Energy  supply 

Area  &  regional  development 

? 

7 

SSA 

600 

650 

Other  income  security 

Social  Security 

28,150 

390,999 

mm 

7 

7 

800 

Federal  law  enforcement 

Central  fiscal  operations 

3,722 

8.710 

11 

2 

7 

7 

IaCDa  i 

1^6 

Conduct  of  foreign  affairs 

43 

1 

7 

300 

Conservation  &  land  management 

1 

>1 

7 

*  Agencies  are  listed  in  the  same  order  as  they  appear  in  Chart  1 ,  in  descending  order  according  to  the  amount  of  R&D  dollars 
each  requested  for  FY  1998. 

**  These  dollars  are  controlled  by  the  Corps  of  Engineers,  an  organization,  that  while  housed  In  DOD,  more  closely  resembles  non- 
DOD  agendas  in  terms  of  its  organizational  structure  and  operations. 

An  examination  of  other  civilian  activities  of  the  federal  government  reveals  a  similar  relationship  between  R&D  and 
S&T. 


As  illustrated  in  Table  2.3  above,  the  set  of  potential  S&T  activities  in  all  civilian  agencies  of  the  federal 
government  appears  to  be  larger  than  the  set  of  R&D  activities  in  these  same  agencies.  The  magnitude  of  their  S&T 
activities  is  hard  to  determine  with  any  precision,  however,  as  for  most  agencies  they  are  not  officially  labeled  S&T. 
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What  is  certain  is  that  several  specific  activities  that  are  widely  believed  to  be  R&D  turn  out  instead  to  be  S&T 
activities  that  fall  outside  the  set  of  activities  officially  designated  as  R&D.  The  best  known  of  these  is  the 
Manufacturing  Extension  Program  at  the  Department  of  Commerce,  which  is  considered  by  many  to  be  R&D,  even 
though  it  has  never  been  designated  as  such. 

CONFLICTING  MODELS 


Figure  2.1  R&D  vs.  S&T 


Military  Activities  Civilian  Activities 


The  chart  above  illustrates  the  two  conflicting  models  that  prevail  in  the  federal  government  of  how  R&D  and 
S&T  relate.  Complicating  this  picture  is  the  fact  that  each  federal  agency  applies  the  definition  of  R&D  differently. 
That  is,  because  federal  agencies  have  different  missions,  the  determination  of  what  constitutes  R&D  in  each  agency 
is  made  from  a  different  perspective.  In  some  instances,  these  differing  perspectives  result  in  agencies  including 
different  activities  in  the  set  of  activities  that  are  designated  as  R&D,  as  described  in  the  example  in  the  previous 
section.  In  other  instances,  this  results  in  differences  among  agencies  as  to  which  R&D  activities  constitute  Basic 


-42- 


Research,  which  constitute  Applied  Research,  and  which  constitute  Development.  It  is  clear  fliat  activities  designated 
as  Basic  Research  by  some  agencies  would  likely  be  designated  as  Applied  Research  by  other  agencies  and  so  forth. 
While  the  extent  of  these  discrepancies  has  never  been  systematically  explored  across  the  federal  government,  these 
differences  have  definitely  had  an  affect  on  the  allocation  of  R&D  resources,  as  these  three  categories  are  among  the 
major  divisions  used  as  bases  for  making  allocations. 

IMPLICATIONS  OF  DEFINITIONAL  DIFFERENCES  FOR  LITERATURE  REVIEW 

Executive  Order  12882  described  the  purpose  of  the  National  Science  and  Technology  Council  (NSTC)  as 
coordinating  “S&T,”  while  the  NSTC’s  1997  Annual  Report  describes  the  purpose  of  the  council  as  coordinating 
“R&D.”  As  demonstrated  above,  S&T  and  R&D  do  not  refer  to  the  same  activities,  so  which  activities  are  the  NSTC 
trying  to  coordinate?  In  a  further  example,  the  charge  to  the  RAND  research  team  for  the  present  project  uses  the 
terms  “research”  and  “R&D”  interchangeably.  As  noted  in  the  first  chart  in  this  section,  however,  “research” 
encompasses  only  two  (i.e.,  Basic  Research  and  Applied  Research)  of  the  three  categories  comprising  the  “Conduct  of 
R&D”  and  only  two  of  the  five  categories  comprising  the  entirety  of  federal  R&D.  So,  what  is  the  focus  of  this 
report?  Examples  such  as  diese  aboimd  throughout  flie  literature  on  federal  R&D. 

The  collective  failure  to  agree  on  the  definition  of  critical  terms  and  then  to  apply  these  definitions  consistently 
has  defeated  and  continues  to  defeat  basic  communication  in  the  federal  R&D  community.  Given  this,  it  is  easy  to 
understand  why  coordination  and  priority  setting  efforts  have  also  suffered. 

Complicating  all  efforts  to  coordinate  and  set  priorities  for  federal  R&D  activities  is  the  general  assumption 
that  the  R&D  activities  of  federal  agencies  resemble  one  another.  While  all  have  been  aggregated  under  the  general 
umbrella  of  federal  R&D  and  all  involve  expenditures  whose  benefits  are  long-term,  the  similarities  diminish  beyond 
this.  For  example,  all  of  NSF’s  R&D  activities  focus  exclusively  on  research,  most  especially  Basic  Research,  to  the 
total  exclusion  of  Development  and  are  conducted  extramurally  using  grants.  The  nature  of  HHS’s  R&D  is  similar, 
although  it  does  some  Development  and  many  of  its  R&D  dollars  are  spent  intramurally  in  federal  laboratories.  In 
contrast,  over  half  of  NASA’s  and  DOD’s  R&D  activities  are  Development  and  most  are  conducted  extramurally 
using  contracts.  The  vast  majority  of  DOE’s  R&D  activities  are  conducted  extramurally  using  contracts,  but  less  than 
a  third  of  them  are  spent  on  Development  activities.  The  vast  majority  of  USDA’s  and  DOC’s  R&D  activities  involve 
research,  with  an  emphasis  on  Applied  Research,  and  most  are  conducted  intramurally  in  federal  laboratories. 

Together  these  seven  agencies  annually  spend  approximately  95  percent  of  all  federal  R&D  funds,  yet  clearly  none  of 
them  approach  R&D  and  its  management  in  the  same  manner.  Given  this,  it  is  understandable  why  the  search  for 
models  of  allocating  federal  R&D  funds  has  not  borne  more  fruit.  This  is  especially  so  for  models  concerning  the 
allocation  of  these  funds  by  field  of  science  and  engineering  as  these  categories  have  no  relevance  to  Development 
activities,  which  account  for  close  to  60  percent  of  all  federal  R&D  activities. 
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DATA  AND  INFORMATION  SOURCES  SUPPORTING  PRIORITY-SETTING  AND  COORDINATION 
ACROSS  THE  DIFFERENT  FIELDS  OF  SCIENCE 

One  of  tiie  central  requirements  for  an  effective  benefits  assessment  regime  is  access  to  databases  of 
sponsored  research  which  may  be  linked  to  databases  of  research  outputs.  (Kostoff,  p.  33)  This  section 
examines  sources  of  information  about  research  and  development  (R&D)  conducted  by  the  federal 
government.  It  first  discusses  those  sources  containing  information  across  agencies  of  the  federal 
government  and  then  those  within  specific  agencies. 

Databases  and  Information  Sources  Across  Agencies 

There  are  three  different  types  of  data  sources  containing  information  about  R&D  activities  across 
agency  lines  in  the  federal  government.  Until  quite  recently,  none  of  these  were  especially  useful  in 
supporting  government-wide  priority  setting  or  coordination. 


Budget  Data 

The  first  type  of  data  source  contains  budget  information  at  a  highly  aggregated  level  (that  is,  with  no 
information  on  specific  tasks  or  awards).  The  main  one  of  these  is  the  report  prepared  by  the  OMB  for 
inclusion  in  the  annual  Budget  of  the  United  States  Government.  This  report  consists  of  a  chapter  in  the 
Analytical  Perspectives  volume  of  the  federal  budget  and  has  been  prepared  annually  beginning  with  the  FY 
1996  budget.  The  dollar  amounts  it  contains  are  "Outlays"  that  do  not  match  the  federal  R&D  baseline 
numbers  in  the  main  budget,  which  are  in  "Budget  Authority."  An  overview  section  on  R&D  in  the  main 

portion  of  each  year’s  budget  complements  this  ch^ter. 

Another  source  of  highly  aggregate  information  is  the  Federal  Funds  for  Research  and  Development 
report  series  that  has  been  compiled  and  published  annually  since  the  1950s  by  the  NSF.  This  report 
provides  a  retrospective  view  of  federal  R&D  activities.  The  dollar  amounts  it  presents  are  “Obligations,” 
rather  riian  “Budget  Authority,”  so  they  do  not  match  those  in  the  official  R&D  baseline  presented  in  the 

federal  budget. 

A  second  type  of  data  source  consists  of  transactional  or  “bookkeeping”  databases,  foremost  of  which 
are  the  Federal  Assistance  Awards  Data  System  (FAADS)  and  the  Federal  Procurement  Data  System 
(FPDS).  The  former  tracks  spending  on  all  federal  grants,  cooperative  agreements,  and  other  non¬ 
procurement  activities,  while  the  latter  tracks  spending  on  all  contracts  with  a  value  of  over  $25  thousand. 
Neither  of  these  is  used  to  track  R&D  activities  specifically,  but  rather  contain  information  on  all  federal 
activities,  be  they  R&D  or  otherwise.  In  contrast  to  the  data  sources  containing  only  budget  information, 
these  sources  contain  only  dis^gregated  data  on  specific  awards,  none  of  which  are  connected  with  higher- 
level  program  or  budget  information. 
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FEDRIP 

The  Federal  Research  in  Progress  (FEDRP)  database  attempts  to  present  a  more  comprehensive 
picture  of  federal  R&D  activities.  This  database  was  created  in  1946  as  the  Smithsonian  Science  Information 
Exchange  and  has  been  maintained  by  the  National  Technical  Information  Service  (NTIS)  at  the  Department 
of  Commerce  since  1981.  Reporting  to  FEDRIP  is  voluntary,  and  thus  it  is  nowhere  near  a  complete  picture 
of  federal  R&D  spending  or  activity.  FEDRIP  contains  no  information  on  defense-related  science  R&D  and 
covers  only  a  portion  of  civilian  R&D,  often  with  quite  dated  records.  The  data  in  FEDRIP  come  primarily 
from  several  agency-specific  sources  -  USDA’s  Current  Research  Information  System  (CRIS),  HHS*s 
Computer  Retrieval  of  Information  on  Scientific  Projects  (CRISP),  NSF's  Science  and  Technology  System 
(STIS),  and  DVA*s  R&D  Information  System  (RDIS)  databases-that  are  discussed  below.  Several  of  these 
agency-specific  databases  emphasize  the  receipt  and  processing  of  applications  for  R&D  funding  and 
consequently  are  individually  accessible  in  some  form  via  the  web.  None  is  connected  to  the  same  agency’s 
R&D  budget  information,  however,  so  there  is  no  way  to  relate  plans  to  outcomes. 

None  of  the  data  assembled  to  fonn  FEDRIP  is  baselined  to  the  federal  R&D  budget.  As  a  result, 
users  of  FEDRIP  cannot  determine  what  portion  of  the  federal  R&D  portfolio  FEDRIP  actually  tracks.  By 
carefully  matching  the  individual  records  in  FEDRIP  to  the  overall  contents  of  the  federal  R&D  portfolio, 
however,  it  appears  that  FEDRIP  contains  data  on  about  20%  of  all  federal  R&D  activities.  Just  recently, 
NTIS  was  been  marked  for  dissolution  by  the  Secretary  of  Commerce.  This  now  appears  to  have  passed,  but 
the  future  of  FEDRIP  is  remains  uncertain. 

RaDiUS 

The  newest  information  source  on  federal  R&D  spending  is  the  RaDiUS  (Research  and  Development 
in  the  United  States)  database,  developed  by  RAND  in  cooperation  with  NSF.  RaDiUS  is  the  first 
information  system  that  systematically  connects  the  highly  aggregated  budget  data  on  federal  R&D  with  the 
disaggregated  information  on  individual  R&D  tasks  and  awards  to  provide  a  complete  picture  of  all  federal 
R&D  activities  and  to  connect  inputs  with  outputs.  To  achieve  this,  RaDiUS  contains  five  interconnected 
levels  of  progressively  more  detailed  data  on  federal  R&D.  The  least  detailed  level  of  RaDiUS  contains 
information  on  the  24  agencies  that  control  and  disburse  all  R&D  dollars  spent  by  the  federal  government. 
The  amounts  carried  in  this  level  of  RaDiUS  are  identical  to  those  presented  in  the  federal  budget  for  Basic 
Research,  Applied  Research,  and  Development.  These  three  "stages”  of  R&D  collectively  constitute  the 
"Conduct  of  R&D"  and  comprise  the  official  "baseline"  of  the  federal  R&D  portfolio.  RaDiUS  does  not 
track  funds  spent  on  the  construction  and  rehabilitation  of  federal  R&D  facilities  or  the  purchase  of  major 
R&D  equipment.  When  funds  for  these  two  activities  are  combined  with  those  for  the  Conduct  of  R&D,  the 
result  is  the  total  budget  for  the  federal  R&D  enterprise. 

RaDiUS  harvests  data  on  federal  R&D  activities  from  information  already  gathered  by  the  federal 
government,  even  if  they  have  not  traditionally  been  viewed  as  relevant  to  the  tracking  of  federal  R&D 
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activities.  For  example,  RaDiUS  includes  information  from  the  Federal  Assistance  Awards  Data  System 
(FAADS)  and  tiie  Federal  Procurement  Data  System  (FPDS),  neither  of  which  has  ever  before  been  used  to 
track  R&D.  The  harvested  data  have  then  been  woven  together  and  augmented  with  information  from 
agency-specific  R&D  databases  and  information  collections  using  common  data  fields  and  codes  to  form  a 
comprehensive  picture  of  federal  R&D.  RaDiUS  has  been  used  by  numerous  federal  agencies  and 
contractors  to  support  R&D  planning  and  coordination  efforts,  leverage  R&D  investments,  and  transfer 

technology. 

Databases  and  Information  Sources  Within  Agencies 

Within  agencies,  there  tend  to  be  two  distinct  sources  of  data  -  those  containing  budget  and  programmatic 
information  generated  for  the  annual  budget  justification  process  and  those  which  track  the  disbursing  of  the  agency’s 
fimds.  The  records  in  fliese  latter  systems  are  not  uniform  or  consistent  across  agencies.  In  many  cases  these  are  not 
databases  at  all,  but  p^er  records.  Generally,  there  are  no  direct  connections  between  these  two  sources  of  data. 

The  major  R&D  agency-specific  award/task  databases  include  DOD’s  TEAMS,  HHS's  Computer 
Retrieval  of  Information  on  Scientific  Projects  (CRISP),  NASA’s  FAC/507s,  DOE’s  R&D  Tracking  System 
(created  at  DOE’s  request  by  RAND),  USDA’s  Current  Research  Information  System  (CRIS),  NSF's  Science 
and  Technology  System  (STIS),  and  Department  of  Veterans’  Affairs  VA's  R&D  Information  System 
(RDIS)  databases. 

Some  of  these  databases  are  used  within  agencies  to  help  interested  parties  leam  more  about  on-going 
R&D  efforts  to  reduce  duplication  and  encourage  coordination  of  R&D  activities.  Given  the  lack  of 
connection  to  agency  budgets  and  program  plans,  it  is  unlikely  that  these  databases  are  widely  used  to 
allocate  funds  or  coordinate  R&D  activities  at  an  aggregate  level.  Most  of  these  agency-specific  R&D 
databases  are  accessible  to  the  general  public  in  some  manner  via  the  web,  while  others  have  highly  restricted 

access. 

Tables  SS 1 .4-  SS 1 .6  presents  a  list  of  federal  data  sources  containing  information  on  science  funding 
and  activities.  None  of  these  sources,  however,  is  devoted  exclusively  to  science  or  “S&P’  activities,  which 
are  virtually  impossible  to  track  given  existing  information.  Furthermore,  the  relationship  between  the  terms 
“R&D”  and  “S&T”  is  problematic.  Althou^  “R&D”  and  “S&T”  are  often  used  as  synonyms,  they  cany 
different  meanings  in  the  defense  and  civilian  agencies,  and  the  term  “S&T’  itself  carries  multiple  meanings. 

Uck  of  awareness  of  this  feet  continually  impedes  efforts  to  improve  planning,  management,  and 
coordination  of  federal  science  activities. 
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Table  SS1.4 

Data  Sources  on  Federal  Science  Activity 


Government-wide  Data  Description 

Sources 


President’s  Budget  (OMB/MAX 
System) 

(Government-Wide  Data 
System) 

Catalog  Of  Federal  Domestic 
Assistance  (CFDA) 
(Government-Wide  Data 
System) 


Federal  Assistance 
Awards  Data  System 
(FAADS) 

(Government-Wide  Data 
System) 

Federal  Procurement  Data 
System  (FPDS) 
(Government- Wide  Data 
System) 


A  public  information  source 
advertising  grants,  cooperative  agreement, 
and  other  non-procurement  awards 
available  from  federal  sources;  contains  no 
information  on  actual  spending  or  R&D 
activity 


Contains  basic  tracking  and 
spending  infonnation  on  all  grants, 
cooperative  agreements,  and  other  non¬ 
procurement  awards  (bodi  R&D  and  non- 
R&D)  made  by  every  federal  agency. 

Contains  basic  tracking  and 
spending  infonnation  on  all  contracts 
(both  R&D  and  non-R&D)  awarded  by 
every  federal  agency  with  a  value  of  over 
$25k. 


Useful  for 

Priority- 

Setting, 

Coordination, 

or  Budget 

Allocation? 

No 


No 


No 


No 


Federal  Research  in 
Progress  (FEDRIP) 


Contains  detailed  information  on  Some 

approximately  25  percent  of  federal  R&D  coordination 
activities  (HHS,  USDA,  NSF,  and  DVA). 


Research  and 
Development  in  United 
States  (RaDiUS) 


Comprehensive  data  system  built 
with  existing  information  that  captures  80- 
90%  of  federal  R&D  activities,  linking 
high-level  budget  information  to 
task/award-specific  spending  and  activity 


Coordination 
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Table  SS1.5 

Data  Sources  on  Federal  Science  Activity 


Agency-Specific  Data  Sources  Description 

(databases) 


Current  Research  Information 
System  (CRIS 

Science  and  Technology  Information 
System  (STIS-NSF) 

Computer  Retrieval  of  Information 
on  Scientific  Projects  (CRISP) 

507  System  (NASA) 

Procurement  and  Assistance 
Data  System  (PADS) 


DOE  R&D  Tracking  System 


NIST/OD 


DVA  RDCC 


A  database  that 
contains  information  on 
ongoing  agricultural,  food  and 
nutrition,  and  forestry 
research  supported  by  USDA 
agencies 

An  electronic 
dissemination  system  that 
provides  access  to  NSF 
publications 

Contains  project 
abstracts  for  the  IMPAC 
data 

Database  tracking 
basic  information  on  contract- 
based  award  and  task 
activities 

Tracks  all  unclassified 
procurements  and  financial 
assistaince  made  directly  by 
DOE  (does  not  detail 
activities  of  National  Labs) 
Contain  award/task 
information  on  activities  of  all 
National  laboratories  and 
comparable  R&D  facilities 
within  DOE’S  purview 
Specially 

prepared  detail  on  R&D 
activities  accessible  via 
RaDiUS  only 

Contains  basic 

information  on  R&D  task  on¬ 
going  within  VA  Medical 
Centers _ _ 


Useful  for  Priority- 
Setting, 

Coordination,  or 
Budget 

Allocation? _ 

Coordination 


Coordination 


Coordination 


No 


No 


Coordination 


No 


No 
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Tabie  SS1.6 

Data  Sources  on  Federal  Science  Activity 


Agency-Specific  Data  Sources 
(budget  justification 

Description 

Useful  for  Priority- 
Setting,  Coordination, 
or  Budget  Allocation? 

ACF  CFO 

Budget  information 

Budget  Allocation 

AHCPR  CFO 

Budget  infonnation 

Budget  Allocation 

AID  CFO 

Budget  infonnation 

Budget  Allocation 

CDC  CFO 

Budget  infonnation 

Budget  Allocation 

DEb  CFO 

Budget  infonnation 

Budget  Allocation 

DOD  CFO 

Budget  infonnation 

Budget  Allocation 

DOE  CFO 

Budget  infonnation 

Budget  Allocation 

DOI  CFO 

Budget  infonnation 

Budget  Allocation 

DOJ  CFO 

Budget  infonnation 

Budget  Allocation 

DOT  CFO 

Budget  information 

Budget  Allocation 

DVA  CFO 

Budget  infonnation 

Budget  Allocation 

EPA  CFO 

Budget  infonnation 

Budget  Allocation 

FDA  CFO 

Budget  infonnation 

Budget  Allocation 

HCFA  CFO 

Budget  information 

Budget  Allocation 

HRSACFO 

Budget  infonnation 

Budget  Allocation 

MMCCFO 

Budget  information 

Budget  Allocation 

NASA  CFO 

Budget  infonnation 

Budget  Allocation 

NIHCFO 

Budget  infonnation 

Budget  Allocation 

NIST/OD 

Budget  information 

Budget  Allocation 

NOAA  CFO 

Budget  infonnation 

Budget  Allocation 

NRCCFO 

Budget  information 

Budget  Allocation 

NSF  CFO 

Budget  information 

Budget  Allocation 

Smithsonian  CFO 

Budget  information 

Budget  Allocation 

USDA  CFO 

Budget  infonnation 

Budget  Allocation 

uses  CFO 

Budget  information 

Budget  Allocation 
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special  ISSUES  SECTION  2.  BENEFITS  ASSESSMENT  IN  THEORY  AND 

PRACTICE 


The  contribution  to  society  from  investment  in  fundamental  science  has  generally  been  assumed  to  be 
sizeable.  Finding  its  most  significant  formal  policy  statement  in  the  Bush  report  (Bush,  1945,)  it  had  been  an 
operative  principal  for  federal  funding  of  R&D  for  many  years.  For  the  purposes  of  resource  allocation,  however,  a 
better  understanding  of  the  returns  to  fliat  investment  and  the  nature  of  the  relationship  between  inputs,  outcomes, 
and  social  benefits  would  be  highly  desirable.  The  present  discussion  will  briefly  survey  means  for  assessing 
benefit,  the  uses  to  which  such  methods  are  put  in  practice  for  the  purpose  of  priority  setting,  and  a  discussion  of 
data  sources  on  federal  R&D  potentially  available  for  budget  coordination  and  priority  setting  purposes. 

HOW  BENEFITS  TO  SOCIETY  ARE  DETERMINED 


Methods  to  measure  the  performance  and  quantify  the  benefits  ensuing  from  fundamental  science,  when 
considered  from  the  standpoint  of  their  objects  of  study,  generally  fall  into  three  classes.’  The  first  is  to  take  an 
assftt-oriented  approach  that  is  only  a  bit  more  illuminating  than  to  make  general  assertions  of  the  value  of  scientific 
research.  While  not  making  any  strong  statements  on  actual  causal  processes,  the  implicit  argument  is  that  one  may 
compare  R&D  efforts  by  assessing  the  value  of  die  research  assets  they  generate:  size  and  nature  of  research 
facilities,  state  and  vintage  of  available  equipment,  numbers  of  graduates  by  field  and  specialty,  etc.  This  asset- 
oriented  measure  is  largely  focused  on  measuring  system  intermediates:  what  tools  are  put  in  place  as  the  result  of 
expenditures  under  die  federal  research  portfolio?  The  implicit  proposition  is  that  the  more  such  assets  are 
available,  the  more  research  may  be  supported  which,  in  turn,  suggests  a  greater  likelihood  of  providing  a  positive 
effect  on  the  pursuit  of  societal  goals. 

There  are  two  additional  approaches  that  measure  results  more  directly  in  an  attempt  to  speak  to  the  question 
of  ultimate  value  of  the  conducted  research.  The  first  focuses  on  the  formal  documented  output,  of  the  research 
process,  principally  scientific  papers,  in  bibliometric  analyses.  These  methods  include  publication  counts,  citation 
analysis,  and  patent  analysis.®  At  its  best,  bibliometrics  attempts  to  measure  the  linkages  from  one  article  to 

’  There  is  a  large  academic  literature  on  the  theory  and  practice  of  this  form  of  measurement  A  more  general 
bibliography  and  analysis  may  be  found  in  (Popper,  1999.)  Kostoff(1997)  provides  an  exhaustive  survey  and  classifies 
methodologies  less  by  their  objectives  than  by  technique  employed:  retrospective  methods  (e.g.  case  studies,)  qualitative 
methods  (e.g.  peer  review,)  and  quantitative  methods  (e.g.  bibliometrics;  economic  analyses.) 

8  For  detailed  explanations  of  bibliometrics,  see:  van  Raan,  A.  F.  J.,  1993.  “Advanced  Bibliometric  Methods  to  Assess 
Research  Performance  and  Scientific  Development;  Basic  Principles  and  Recent  Practical  Applications,”  review  report  based  on 
invited  paper.  University  of  Leiden  Report  CWTS-93-05,  August;  Cozzens,  Susan  E.,  1989.  “Literature-Based  Data  in  Research 
Evaluation:  A  Manager’s  Guide  to  Bibliometrics,”  report  to  the  National  Science  Foundation;  Narin,  Francis,  Dominic  Olivastro, 
and  Kimberly  Stevens,  1994.  “Bibliornetricsmteory,  Practice  and  Problems.”  Evaluation  Review,  Vol.  18,  No.  1,  February,  pp. 
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another  to  measure  the  intellectual  connections  between  organizations  and  the  linkages  between  subject  areas. 
Citation  analysis  can  identify  a  group  of  articles  that  are  cited  together.  These  data  can  help  construct  a  “cognitive 
map”  of  scientific  research  activities,  identifying  where  science  is  adding  value  to  society.  These  methods  have 
individual  benefits  and  drawbacks,  but  as  an  approach  for  measurement  they  are  hard-pressed  by  themselves  to 
make  the  connection  between  the  outputs  they  measure  and  the  influence  of  these  outputs  on  the  creation  of  social 
goods  except  in  the  broadest  possible  sense.  They  are,  however,  an  important  key  to  achieving  better  understanding 
of  the  process  that  yields  scientific  outcomes  and  ultimate  social  benefit. 

Yet  a  third  class  seeks  to  measure  outcomes  or  results  effected  through  and  by  the  results  of  scientific 
research.  Two  fundamental  approaches  are  used.  The  crafting  of  case  studies  and  retrospective  analyses  is  largely  a 
narrative  method  and  by  definition  narrowly  focused  on  following  specific  threads  of  benefit.  This  approach  was 
used  in  the  Air  Force  “Hindsight”  study  and  the  NSF’s  TRACES  project  and  its  follow-on  as  well  as  the  more  recent 
effort  by  NSF’s  Directorate  of  Engineering.^  Again,  by  definition,  this  methodology  is  very  useful  in  improving  our 
imderstanding  of  the  process  of  discovery  and  innovation  and  in  illuminating  lessons  for  priority  setting,  but  it  is 
also  resource  intensive  and  necessarily  takes  place  long  after  particular  decisions  on  allocation  have  been  made. 

The  application  of  economic  techniques  and  concepts  to  results-oriented  assessment  yields  methodologies 
that  have  been  used  to  search  for  quantifiable  effects  ensuing  from  scientific  research.  Yet,  most  such  approaches 
were  developed  to  answer  specific  questions.  Often  in  their  specification  they  cannot  capture  fully  the  intricacies 
involved  in  the  conduct  of  science  nor  the  nature  of  scientific  research  as  an  industrial  input.  Such  studies  almost 
always  use  one  of  the  following  as  the  metric  of  performance:  productivity  growth,  increase  in  national  income,  or 
cost  versus  benefit — ^particularly  improvement  in  social  welfare  as  measured  by  changes  in  consumer  and  producer 
surpluses.  Even  so,  despite  their  limitations,  economic  measures  are  used  to  describe  the  contributions  of 
fundamental  science.  The  methodologies  most  often  employed  include  production  function  analysis  and  social  rates 
of  return  analysis.  " 

Production  function  analysis  is  a  mathematical  mapping  of  the  inputs  of  a  production  process  to  its  outputs. 

A  general  finding  fi-om  such  studies  is  that  returns  to  R&D  in  general  are  high,  but  the  returns  to  federal 


65-76;  van  Raan,  A.  F.  J.,  1993.  “Advanced  Bibliometric  Methods  to  Assess  Research  Performance  and  Scientific  Development: 
Basic  Principles  and  Recent  Practical  Applications,”  review  report  based  on  invited  paper.  University  of  Leiden  Report  CWTS- 
93-05,  August. 

^  Illinois  Institute  of  Technology  Research,  “Technology  in  Retrospect  and  Critical  Events  in  Science”,  1968;  Director 
DDR&E,  “Project  Hindsight”,  DTIC  No.  AD495905, 1969;  Battelle  Columbus  Laboratories,  “Interactions  of  Science  and 
Technology  in  the  Innovative  Process:  Some  Case  Studies”,  1973. 

^^ere  are,  of  course,  alternative  ways  to  categorize  studies  in  the  aggregate  assessment  literature.  Hertzfeld  (1990) 
offers  a  useful  classification  scheme  for  studies  on  the  contributions  made  by  fundamental  research.  Rather  than  utilize 
methodology  as  an  organizing  principle,  he  establishes  five  categories  to  group  studies  by  issue:  (1)  research  output  at  the 
sectoral  level,  (2)  measurement  of  social  welfare,  (3)  evaluation  of  emerging  technologies,  (4)  research  infrastructure 
development,  and  (5)  research  stmcture  efficiency. 
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expenditures  on  R&D  are  low."  Yet,  in  many  ways  federal  research  presents  greater  problems  for  measurement  and 
benchmarking  than  does  private  R&D.  Much  federally  fimded  research  is  directed  to  areas  where  there  is  only  a 
limited  market  at  best.  Further,  given  the  types  of  data  available,  the  returns  that  result  from  most  calculations  must 
be  interpreted  as  average  rather  than  marginal  rates.  From  the  policy  perspective,  this  means  we  cannot  be  certain  of 
the  implications  to  draw  from  this  aggregate  analysis  for  what  the  effect  of  an  additional  dollar  of  expenditure  on 
research  might  be.  The  cost/benefit  framework  itself  may  itself  be  too  restrictive,  failing  to  capture  the  many  types 
of  benefit,  which  may  be  derived  from  publicly-funded  basic  research.  The  true  effect  of  such  outlays  may  well  be 
indirect,  affecting  productivity  through  changing  the  returns  to  private  research  and  development  rather  than  directly 
as  a  result  of  the  specific  research  project. 

Social  rates  of  return  analysis  departs  from  the  narrow  specifications  of  a  production  function  and  seeks  to 
determine  the  sum  of  benefits  accrued  from  changes  in  the  knowledge  base  and  compares  these  benefits  to  the  cost 
of  investment.  These  benefits  are  most  often  measured  as  changes  in  consumer  and  producer  surplus.  This  social 
benefit  may  be  considerably  greater  than  the  private  benefit  taken  in  the  form  of  profit.  As  a  practical  matter,  such 
studies  involve  selecting  a  sample  of  specific  irmovations  upon  which  to  perform  these  calculations.  This  is  both 
expensive  and  subject  to  unintentional  or  unavoidable  bias  of  one  sort  or  another  in  the  selection  process.  Further, 
the  social  rate  of  return  calculated  by  such  means  is  not  directly  comparable  to  the  internal  rates  of  return  calculated 
for  private  investment  projects.  Nevertheless,  studies  in  this  area  have  found  a  very  high  return  to  investment  in 
basic  research.  In  a  much  publicized  result,  one  paper  found  a  28  percent  return  to  investment  in  academic 
research.l2  yet,  here  again,  the  principal  outcome  from  such  studies  is  to  suggest  the  need  for  other  analyses  to 
address  specific  aspects  of  concern. 

UTILIZATION  OF  BENEFITS  MEASURES  IN  PRIORITY  SETTING  AND  BUDGET  COORDINATION 

Who  are  the  customers  of  such  methods  for  calculating  benefits  from  scientific  research?  Within  the 
government  it  is  perhaps  not  surprising  that  the  National  Science  Foundation  was  an  early  and  major  funder  of  such 
work.  However,  it  is  only  recently  ftat  that  agency  and  others  have  begun  to  consider  how  the  methods  and  findings 

Some  summaries  of  the  studies  which  support  these  stylized  facts  are  Zvi  Griliches,  1973,  “Research  Expenditures 
and  Growth  Accounting,”  in  Science  and  Technology  in  Economic  Growth,  B.  R.  Williams,  ed.  (New  York:  Wiley)  pp.  59-83; 
Zvi  Griliches,  1980.  “Returns  to  Research  and  Development  Expenditures  in  the  Private  Sector,”  in  New  Developments  in 
Productivity  Measurement  and  Analysis,  J.  W.  Kendrick  and  B.  N.  Vaccara  (eds.)  (Chicago:  University  of  Chicago  Press)  pp. 
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produced  by  such  efforts  may  be  used  operationally  to  assess  performance  and  inform  decisions  over  priority  in 
funding. 

One  principal  potential  customer  of  such  information  is  Congress.  Whether  it  acts  as  the  originator  of 
information  requests  for  the  purposes  of  furthering  its  own  process,  or  is  targeted  as  the  ultimate  audience  for 
assessments  produced  for  its  benefit  by  the  agencies  coming  before  its  committees  for  funding.  Congress  would  like 
to  have  better  means  for  determining  the  results  ensuing  from  federal  funding  of  R&D.  The  most  recent  reiteration 
of  this  desire  from  the  Congressional  standpoint  is  found  in  the  recent  report  of  the  House  Science  Committee  (U.S. 
Congress,  1998).  However,  this  document  reinforces  that  there  is  still  a  lack  of  any  well-documented  connection 
between  ultimate  results  and  ex  ante  calculations  of  benefit.  This  provides  in  large  part  the  motivation  for  continued 
support  to  curiosity-driven  research  (p.  13).  To  the  extent  Congress  should  seek  to  measure  success  in  this  areas  it 
should  be  done  on  an  aggregate  portfolio  and  not  on  individual  research  project  basis  (p  22.) 

Another  body  which  has  considered  broad  application  of  performance-based  measures  throughout  the  federal 
government  is  the  NSTC,  That  body  has  issued  a  list  of  performance  measures  for  function  250  (R&D)  activities: 

•  the  federally  funded  research  is  to  be  of  “high  quality” 

•  encourage  setting  aside  80  percent  or  more  for  peer-reviewed  competition 

•  encourage  assessment  of  programs  by  external  bodies 

•  build  and  operate  major  scientific  facilities  efficiently 

•  and  meet  federal  time  and  cost  guidelines  for  facility  upgrades  and  minimal  down  time  (NSTC,  1998;  p. 

30). 

As  may  be  quickly  seen,  these  are  in  reality  guidelines  for  conduct  and  measures  attuned  to  the  first  category  foimd 
in  the  literature:  asset-based  measures.  There  is  no  discussion  of  applying  measures  in  terms  of  either  direct  outputs 
or  indirect  outcomes. 

The  work  of  the  National  Academy  of  Sciences  provides  a  further  example  of  this  approach.  This  body,  and 
in  particular  its  Committee  on  Science  Engineering  and  Public  Policy  (COSEPUP,)  has  suggested  assigning  priority 
to  areas  of  science  in  such  a  way  that  the  U.S.  is  positioned  among  the  leaders  in  all  fields.  Hence,  there  is  a  natural 
gravitation  toward  enabling  more  effective  international  benchmarking  as  a  purpose  for  assessing  benefits.  The 
general  guidelines  suggested  by  COSEPUP  are:  1)  What  is  the  U.S.  position  in  a  field?  2)  What  key  factors 
determme  relative  position?  and  3)  What  is  the  trend  for  relative  position  in  the  near  and  long  term?  (NAS  1998,  p. 
15)  Due  to  their  aggregate  nature,  these  yardsticks  do  not  necessarily  track  well  with  those  of  potential  use  to 
mission  agencies.  Further,  in  application,  most  actual  measures  employed  again  appear  to  be  of  the  asset-accounting 
type.  In  a  pilot  study  of  materials  science  intended  to  implement  this  type  of  assessment,  COSEPUP  utilized  the 
following  measures: 

•  National  imperatives  as  an  influence  on  S&T 

•  Iimovation 

• 


Major  facilities 
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•  Centers 

•  Human  resources 

•  Funding  (NAS  1 998,  p.27.) 

Most  of  these  are  a  matter  of  weighing  of  assets.  The  first  and  especially  the  second  are  more  matters  of 
interpretation.  Only  the  latter  can  truly  be  considered  to  be  a  measure  of  either  direct  outcome  or  indirect  results. 

At  the  agency  level,  the  DOE  Office  of  Science  Strategic  Plan  lists  a  series  of  success  indicators  for  each  of 
its  five  main  goal  areas  (e.g.:  “photochemical  systems  that  hold  promise  for  economical,  highly  efficient  solar 
cells.”)  The  indicators  are  outcome-oriented  but  seem  to  be  of  a  checklist-type,  attuned  to  achieving  particular 
milestones  and  not  quantitative  in  nature.  They  do  not  seek  to  track  direct  benefit  back  to  specific  R&D  project 
outcomes  (DOE,  1999.) 

Documents  on  the  performance  assessment  process  in  use  by  NSF  point  to  heavy  reliance  on  external 
assessment.  Attempts  are  being  made  to  shift  from  a  somewhat  ad  hoc  basis  to  a  more  formal  procedure  that  will 
provide  a  common  format  to  the  review  process  that  extends  across  NSF.  In  addition,  begirming  in  1998,  aimual 
reports  to  the  Director  have  been  required  from  NSF  units  using  a  variety  of  indicators  and  data  series  to 
demonstrate  effect  (NSF,  September  1997.) 

Results  from  NSF  awards  appear  over  time.  Thus,  in  assessing  performance  toward  die  outcome 
goals  during  a  given  year,  NSF  must  look  carefully  at  (1)  noteworthy  achievements  of  the  year  based 
on  NSF  awards,  (2)  the  ways  in  which  projects  collectively  affect  progress,  and  (3)  erqpectations  for 
future  performance  based  on  the  current  set  of  awards.  (NSF,  Jan.  99,  p.  15) 

This  states  the  first  of  the  two  main  problems  in  assessing  benefits  and  using  the  results  in  priority  planning 
at  a  program,  directorate,  or  even  agency  level:  it  is  difficult  to  establish  causal  linkages  between  research  and 
outcomes  over  time.  However,  in  this  regard  NSF  has  explicitly  addressed  the  other  problem  that  usually  bedevils 
the  practical  use  of  this  type  of  assessment.  It  has  attempted  to  spell  out  the  goals  it  seeks  to  achieve  from  its 
endeavors.  The  NSF  GPRA  performance  plan,  as  amended,  contains  a  series  of  goals  and  levels  of  performance. 
For  the  more  difficult,  outcome-oriented  of  the  goals  (e.g.,  “Outcome  Goal  2:  Connections  between  discoveries  and 
their  use  in  service  to  society”,)  the  principal  means  of  assessment  spears  to  be  reliance  on  expert  judgement  and 
external  directorate  advisory  panels.  As  such,  the  assessment  is  principally  qualitative.  Those  goals  that  are  more 
management  or  facilities  oriented  naturally  find  an  easier  time  in  stating  quantitative  measures  of  output. 

The  academic  literature  continues  to  suggest  new  means  for  assessing  benefit  and  applying  to  considerations 
of  priority.  As  an  example,  an  options  approach  has  begun  to  receive  some  attention  (Vonortas,  1999.)  The 
intention  is  to  formally  reduce  uncertainty  through  a  phased  decision  process  that  might  also  more  easily  support 
handing  off  of  funding  responsibilities  from  the  public  to  the  private  sector  as  technology  opportunities  become 
more  apparent.  Yet,  an  examination  of  the  agency-generated  literature  seems  to  bear  out  the  conclusions  arrived  at 
by  Kostoff  (1997)  in  his  exhaustive  Handbook  of  Research  Impact  ^sessmerA: 
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While  the  methods  used  in  the  performance  of  research  continually  advance  the  state-of-the-art,  the 
methods  used  for  its  identification  and  selection  have  changed  little  in  decades.  In  evaluation  and 
assessment  of  existing  and  completed  research,  not  only  have  the  methods  in  practice  changed  little 
with  time,  but  the  numbers  of  organizations  which  use  any  but  the  most  rudimentary  methods  also 
remain  a  handful.  While  the  scientific  and  social  science  literatures  abound  with  advanced 
methodologies  for  identifying  and  selecting  new  research,  managing  existing  research,  and  evaluating 
and  assessing  research  retrospectively,  the  implementation  of  these  methods  by  the  research 
sponsoring  community  remains  minimal  (p.4.) 


Yet,  the  difficulties  in  applying  the  measurement  means  developed  by  academics  should  not  be 
underestimated.  Questions  of  measurement  revolve  around  certain  conceptual  issues  of  model  specification,  data, 
and  policy.  All  approaches  to  measurement  utilize  models  which  are  at  best  approximations  of  a  technical 
relationship.  In  the  context  of  these  models,  several  general  issues  recur  irrespective  of  method:  (Popper,  1999) 

•  Basic  research  is  conducted  in  a  wide  range  of  fields.  The  mechanisms  for  integrating  research  findings  with 
practical  development  work  may  differ  considerably  fi-om  field  to  field. 

•  All  the  surveyed  approaches  carry  an  implicit  assumption  of  linearity.  Yet,  it  is  precisely  the  non-linearities  and 
interdependencies  which  are  the  hallmark  of  the  innovative  systeni  and  make  the  attribution  of  causality  so 
difficult. 

•  There  are  qualitative  differences  between  knowledge  as  an  input  and  the  more  traditional  inputs  of 
capital  and  labor.  The  proxies  we  use  for  the  former  are  less  satisfactory  measures  of  value  than  those 
used  to  measure  the  latter  two. 

•  Concrete  and  formalized  research  findings  are  the  most  obvious  product  of  basic  research.  There  are 
other  less  formal  outputs  that  may  have  as  great  an  effect  on  economic  and  social  goals.  These 
include: 

-  Training  in  scientific  fundamentals. 

-  Expertise  and  experience. 

-  Network  formation. 

-  Experimental  design  and  protocols. 

-  Increased  number  of  guesses  about  nature  and  greater  chance  for  serendipity. 

-  An  option  for  future  action. 

-  Negative  results. 

-  Intangible  benefits. 

This  underscores  a  need  to  examine  the  assumptions  underlying  any  particular  method  for  measuring 
performance.  It  also  raises  issues  of  data  if  benefits  assessment  is  to  be  seriously  attempted. 
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appendix  A.  FULL  TEXT  DESCRIPTIONS  OF  AGENCY  PRIORITY  SETTING 
ACTIVmES  DESCRIBED  IN  THE  LITERATURE 


Department  of  Defense 

Literature  on  priority  setting  in  the  Department  of  Defense  (DoD)  is  scarce.  Nonetheless,  in  addition  to  the 
1991  OTA  report,  a  few  documents  were  found  regarding  priority  setting;  fliese  included  several  DoD  annual  reports 
and  an  S&T  strategic  plan  as  well  as  a  RAND  study  on  “Priority  Setting  and  Strategic  Sourcing  in  die  Naval 
Research,  Development  and  Technology  Infiastructure,”  (Saunders,  1995). 

According  to  Chapter  16  of  DOD’s  1997  Annual  Report,  planning  for  Science  and  Technology  occurs  in 
the  office  of  the  Director  of  Defense  Research  and  Engineering  (DDR&E).  DDR&E  looks  to  the  NSTC  and  the 
Joint  Chiefs  of  Staffs  Joint  Vision  2010  for  guidance.  The  S&T  plans  are  created  in  two  ways.  First,  overall  future 
needs  are  set  by  program  planners;  tiien,  planners  try  to  match  technology  with  program  goals.  If  current  technology 
cannot  meet  future  goals,  strategies  are  created  to  develop  new  technologies.  Second,  planners  may  also  start  fixim  a 
possible  scientific  opportunity  and  tailor  programs  to  explore  that  opportunity.  In  addition  to  tiiese  guiding 
principles,  the  DoD  S&T  program  sets  forth  plans  for  the  Joint  Warfighting  S&T  Plan  (JWSTP),  Defense 
Technology  Area  Plan  (DTAP),  DoD  Basic  Research  Plan,  and  various  plans  of  die  Military  Departments  and 
Defense  Agencies. 

In  its  Basic  Research  Plan,  DoD  uses  peer  review  and  competition  to  achieve  is  objectives.  In  soliciting 
new  ideas,  DoD  uses  the  broad  agency  announcement  process,  electronic  media,  and  other  mechanisms  to  reach  the 
scientific  community.  Also,  the  Deputy  Under  Secretary  of  Defense  for  Science  and  Technology  establishes 
Technology  Area  Reviews  and  Assessments  (TARA)  as  an  oversight  function  to  assess  the  quality  of  the  research 
programs.  The  TARA  review  teams  consist  of  technical  experts  fi-om  academia,  industry,  and  not-for-profit  research 
organizations.  These  teams  evaluate  the  programs  for  quality,  for  advances  in  leading  the  state-of-the-art  in  research 
areas,  and  for  their  scientific  vision. 

Within  the  DDR&E  operates  the  Defense  Advanced  Research  Projects  Agency,  DARPA.  Created  in  1958, 
DARPA’s  mission  is  to  “...develop  ‘revolutionary’  technologies  that  can  make  a  significant  impact  on  the  future  of 
the  United  States’  defense  posture,  and  to  ensure  that  those  technologies  effectively  enter  the  appropriate  forces  and 
supporting  industiy  base.’  As  OTA  notes,  priority  setting  within  DARPA  is  very  different  from  that  at  other  DOD 
research  agencies.  Project  managers  “are  not  attempting  to  maintain  strength  across  a  field,  rather  they  are  funding 
good  ideas  that  are  on  the  forefront  of  technology  development  to  meet  desired  objectives.”  (p.  105)  DARPA 
program  managers  are  guided  in  decision  making  by  recommendations  established  in  the  JWSTP  and  the  more 
detailed  DTAP.  The  JWSTP  and  DTAP  are  drafted  as  a  result  of  direct  interaction  in  a  series  of  meetings  between 
technical  area  experts,  scientists,  the  portfolio  manager  (DARPA)  and  the  intended  consumer  --  the  military 
services.  These  meetings,  in  turn,  are  informed  by  a  set  of  Joint  Warfighting  Capability  Objectives  (JWCOs)  issued 
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by  the  Joint  Chiefs  in  concert  with  OSD  and  service  science  and  technology  planners  and  decision  makers. 
Individual  JWCOs  are  then  assessed  by  teams  assigned  to  each  during  the  TARA  process.  The  practice  ensures  that 
at  least  two-thirds  of  the  TARA  membership  comes  from  outside  DOD.  The  process  results  in  the  drafting  of  the 

JWSTP  and  DTAP  documents. 

As  for  other  DOD  services,  OTA  notes  that  “compared  wifli  basic  research  funding  for  other  research 
agencies,  the  services  show  remarkably  little  fluctuation  in  allocated  finds,  adjusted  for  inflation.”  (p.  1 1 1)  Reports 
on  other  services’  priority  setting  activities  are  limited. 

In  1995,  the  Department  of  the  Navy  (DON)  contracted  RAND  to  conduct  a  study  on  priority  setting  in 
research  and  development  in  the  Navy.  The  RAND  study  presented  a  framework  intended  to  aid  the  DON  in  setting 
funding  (or  support)  priorities  for  various  lines  of  RD&T.  The  RAND  framework  was  comprised  of  “four  coupled 
(interdependent)  steps:  assembling  a  list  of  RD&T  capabilities,  defining  criteria  or  dimensions  along  which  the 
capabilities  can  be  evaluated,  ranking  the  capabilities  according  to  criteria,  and  translating  the  criteria  rankings  into 
support  priorities.”  (Saunders,  1995)  The  framework,  rather  than  the  application  of  it,  was  regarded  as  the  primary 
output  of  this  project.  To  arrive  at  final  sets  of  RD&T  funding  priorities,  DON  itself  applies  the  framework  to  its 
process. 

According  to  OTA,  much  of  Army  research  is  closely  linked  to  priority  setting  for  all  of  the  R&D  funds  in 
the  Army’s  laboratories  and  institutes.  “Laboratories  in  the  Army  act  independently,  although  they  determine 
priorities  in  relation  to  overall  directives  from  Laboratory  Command.”  Most  are  “industrially  funded”  competing 
for  funds  from  sources  within  and  without  the  Army.  (p.  1 05) 

Department  of  Energy 

According  to  the  OTA  study,  the  Secretary  of  Energy  instituted  a  strategic  plaiming  process  within  DOE  in 
the  late  1980s.  A  National  Energy  Strategy  (NES)  was  designed  to  solicit  input  from  the  offices  within  DOE  and 
from  external  advisors: 

In  the  Office  of  Energy  Research,  programs  such  as  Basic  Energy  Sciences  and  High  Energy  and  Nuclear 
Physics  use  an  “iterative”  process  of  priority  setting-where  ideas  are  proposed  (with  origins  both  within  and 
without  DOE),  feedback  from  the  scientific  conununity  and  other  parts  of  government  are  received,  and  the 
proposal  is  revised.,  .to  determine  goals.  In  particular,  as  national  goals  are  defined  and  new  ideas  arise 
from  either  within  DOE  or  without,  the  program  will  first  consider  them  internally.  If  the  new  initiative 
would  fit  into  the  existing  program  or  complement  it,  then  the  idea  will  be  fielded  to  a  wider  audience. 
Sometimes  this  audience  includes  only  other  parts  of  the  agency.  DOE  may,  however,  hold  public 
workshops  and/or  panel  meetings  to  devise  a  plan  of  action.  (OTA,  1991;  p  1 10) 


to 

This  process  is  exceedingly  labor  intensive.  One  study  seeking  to  explore  new  technical  means  to  make  more 
practicable  the  types  of  interactions  required  for  the  full  operation  of  this  model  may  be  found  in  Lempert  ( 1 998.) 
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In  addition,  each  program  has  an  advisory  panel,  such  as  the  High  Energy  Physics  Advisory  Panel,  an 
external  board  of  scientists.  This  group  and  others  like  it  present  to  DOE  programs  a  set  of  priority  research  areas 
that  deserve  the  agency’s  special  attention. 

According  to  publications  from  the  Department  of  Energy’s  Office  of  Science,  die  office’s  stated  goal  is  to 
pursue  scientific  excellence  in  energy  research  (DOE,  1999).  In  selecting  areas  of  research,  the  Office  of  Science 
emphasized  the  use  of  peer  review  to  evaluate  all  programs.  It  also  stated  that  advisory  boards  play  a  significant  part 
in  its  priority  setting  processes.  As  input  to  the  Office  of  Science  1999  Strategic  Plan,  two  workshops  were  held.  In 
the  first  workshop,  the  participants —  a  group  of  scientists,  technologists,  high  tech  managers  and  science 
communications  people  from  across  all  areas  of  government,  industry  and  academia— established  overarching 
themes  in  the  energy  sciences.  These  themes  included:  improving  fuels  and  power  for  an  energy-efficient  future; 
cleaning  up  the  environment;  preventing  waste  and  pollution,  protecting  human  health;  discovering  the  answers  to 
the  basic  secrets  of  the  universe;  and  providing  the  tools  to  enable  cutting-edge  research.  In  the  second  workshop, 
after  several  debates  and  discussions,  these  themes  were  broken  down  into  five  specific  goals:  establishing  clean  and 
affordable  energy;  protecting  the  planet  by  studying  how  energy  impacts  people  and  the  biosphere,  exploring  matter 
and  energy,  providing  cutting-edge  tools  for  science;  and  pursuing  scientific  excellence.  Under  each  of  these  goals, 
three  objectives  were  created;  and  imder  each  objective,  strategies  were  set  to  attain  the  goals.  For  example,  an 
objective  of  the  Fuel  for  the  Future  program  is  to  “advance  the  science  for  the  development  of  new  and  improved 
sources  of  domestic  fuels.”  The  strategies  listed  under  this  objective  show  that  the  Office  of  Science  will  invest  in 
chemistry  and  material  science  for  energy  conversion,  and  will  conduct  research  in  plant,  microbial  and  solar 

conversion,  and  continue  research  in  the  geosciences. 

The  DOE  R&D  and  Technology  Roadmapping  Framework  in  its  strategic  plan  states  that  “Each  planning 
shall  ensure  that  the  R&D  technology  roadmaps  development  imder  the  direction  of  the  Under  Secretary  are 
fully  integrated  into  die  goals,  objectives,  performance  goals,  and  strategies  of  the  strategic  plan.  Laboratory  input 
will  be  necessary  to  accomplish  this.  However,  this  does  not  mean  that  each  team  needs  to  develop  an  objective  for 
R&D  or  technology.  Other  than  the  Science  mission  area,  R&D  and  Technology  Roadmapping  will  likely  be  a 
cross-cut  topic  and  not  meet  the  budgetary  criteria  for  its  own  objective. 

National  Aeronautics  and  Space  Administration  (NASA) 

NASA  R&D  priorities  are  influenced  more  heavily  by  Congressional  mandate  than  those  of  other  agencies, 
particularly  with  regard  to  large  scale  efforts.  According  to  OTA,  the  Office  of  Space  Science  (OSS)  sets  priorities 
in  conjunction  with  the  budget  process  and  by  selecting  specific  projects.  The  process  is  essentially  bottom  up  with 
project  managers  proposing  new  initiatives.  However,  when  large  missions  are  proposed,  such  as  Space  Station 
Freedom,  top-down  direction  will  determine  the  parameters  of  the  effort.”  (OTA,  1991;  p.  107) 

Priority  setting  in  NASA’s  Office  of  Space  Science  (OSS)  is  a  combination  of  input  from  its  own  internal 
managers,  staff  and  directors,  and  external  actors  like  the  National  Research  Council  (NRC)  and  the  space  science 
community.  In  December  1995,  the  OSS  asked  the  NRC’s  Space  Studies  Board  to  update  the  1991  Bahcall  report 


-58- 


on  scientific  priorities  for  space  astronomy  and  astrophysics.  In  its  goal  to  pursue  scientific  excellence,  the  Office  of 
Science  emphasized  the  use  of  peer  review  to  evaluate  all  programs.  It  also  stated  that  the  advisoiy  boards  will  also 
play  a  significant  part  in  its  priority  setting  processes.  According  to  OTA: 

The  National  Research  Council  (NRC)  plays  a  particularly  strong  advisory  role  for  OSSA,  and  the 
Space  Studies  Board  provides  input  for  most  NASA  basic  research  programs.  The  board  is  unique  at  NRC 
because  it  has  an  institutional  relationship  with  NASA,  i.e.,  NASA  funds  the  board  and  requests  many 
studies,  but  the  board  can  use  these  resources  to  initiate  studies  independently.  In  fact,  the  board  has  been 
able  to  preserve  its  credibility  because  it  has  not  always  agreed  with  NASA,  and  has  openly  disputed  it  on 
some  occasions.  Roughly  every  10  years,  if  events  do  not  call  for  an  earlier  revision,  the  board  writes  a 
strategic  plan  for  every  discipline  in  OSSA.  The  Space  Studies  Board  also  conducts  periodic  reviews  of  the 
programs  and  every  new  mission,  and  other  larger  topics  such  as  ‘‘manned’  vs.  “unmanned’  flight  are 
routinely  studied.  In  addition,  OSSA  has  an  internal  structure  of  advisory  panels.  The  panels  are  usually 
made  up  of  representatives  from  academia,  industry,  Federal  laboratories,  and  other  interested  groups  such 
as  program  managers  from  other  agencies.  They  are  consulted  at  least  once  or  twice  each  year  (sometime 
quarterly)  about  future  directions  for  research  programs.  However,  as  with  NRC,  their  findings  are  never 
binding.  (OTA,  1991;  p  113) 

In  response  to  NASA’s  1995  request,  the  NRC  established  a  Task  Group  on  Space  Astronomy  and 
Astrophysics  (TGSAA).  TGSAA  split  itself  into  four  panels:  Planets,  Star  Formation,  and  the  Interstellar  Medium; 
Stars  and  Stellar  Evolution;  Galaxies  and  Stellar  Systems;  and  Cosmology  and  Fundamental  Physics.  Forty-six 
experts  were  selected  to  serve  on  these  panels.  The  panels  engaged  in  debate,  discussion,  and  a  series  of  ballots  to 
draft  a  series  of  final  priorities.  The  committee  used  several  approaches  in  establishing  the  draft  priorities.  It  tried 
to  “concentrate  on  scientific  objectives  rather  than  the  methods  by  which  the  objectives  are  implemented,  it 
prioritized  scientific  questions  of  significance,  committee  members  were  realistic  to  cost  and  technical  feasibility”. 
The  process  resulted  in  a  report  entitled  A  New  Science  Strategy  for  Space  Astronomy  and  Astrophysics,  The  OSS 
then  used  this  report  as  input  into  its  1997  strategic  plan. 

In  the  1997  Strategic  Plan,  the  OSS  proposed  near-term  (2000-2004)  and  long-term  (2005-2020)  priorities. 
The  plan  established  seven  fundamental  questions  that  form  the  foundation  for  the  office’s  scientific  program.  It 
then  incorporated  the  advice  of  the  NRC  as  well  as  others  in  the  space  science  community  in  establishing 
“Enterprise  Goals”  *,  “Science  Goals”,  and  more  specifically,  “Science  Objectives”.  For  example,  under  the 
category  of  Fundamental  Questions,  OSS  asked,  “how  did  the  Universe  begin  and  what  is  its  ultimate  fate?”  Under 
the  category  of  Enterprise  Goals,  OSS  stated  that  it  wanted  to  “establish  a  virtual  presence  throughout  the  solar 
system,  and  probe  deeper  into  the  mysteries  of  the  Universe  and  life  on  Earth  and  beyond.”  Under  Science  Goals, 
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OSS  stated  that  it  needs  to  “understand  how  structure  in  our  Universe  (e.g.,  clusters  of  galaxies)  emerged  from  the 
Big  Bang.  And,  imder  Science  Objectives,  we  need  to  “observe  the  earliest  structure  in  the  Universe.” 

Department  of  Health  and  Human  Services,  National  Institutes  of  Health 

NIH  is  the  largest  player  in  the  Federal  Government  in  terms  of  dollars  awarded  to  basic  and  applied 
research.  It  is  the  principal  biomedical  research  arm  of  the  U.S.  Department  of  Health  and  Human  Services  (HHS), 
funding  biomedical  and  basic  research  related  to  a  broad  spectrum  of  diseases  and  health  problems  both  in  its  own 
research  facilities  (the  NIH  laboratories)  and  in  external  organizations.  In  terms  of  goal  setting  at  the  agency,  “since 
the  1960s,  the  goals  and  justifications  for  health  research  have  been  fairly  constant-improving  the  health  of  the 
American  people,  curing  particular  chronic  diseases,  and  contributing  to  the  economic  well-being  of  the  Nation  by 
producing  a  healthier  work  force.  However,  particular  emphases  have  shifted.”  (OTA,  1991;  p.  99)  Generally, 
these  shifts  have  occurred  as  new  scientific  developments  have  changed  the  landscape  for  medicine. 

The  priority  setting  process  in  the  twenty-one  National  Institutes  of  Health  consists  of  input  from  Congress 
and  the  science  community  as  well  as  from  NIH  directors  and  staff: 

Each  mctitiitP!  has  an  advisory  council,  which  is  appointed  flirough  HHS  and  is  made  up  of  scientists  and 
lay  people.  Program  officers  must  go  before  the  council  to  present  ideas  for  new  programs,  and  councils 
review  program  balance.  Each  institute  may  also  form  advisory  committees  with  programmatic  foci;  for 
example,  NHBLI  has  six  committees  to  assist  in  specific  fields.  Committees  help  develop  new  initiatives.  It 
should  be  noted,  however,  that  the  council  is  only  advisory,  except  for  its  ability  to  approve  or  disapprove 
grant  applications.  (OTA,  1991;  p.  100) 

According  to  NIH  documents,  the  director  of  each  Institute,  with  the  help  of  NIH’s  national  advisory 
council,  decides  funding  for  grants  and  programs.  The  council  is  mandated  by  Congress  and  is  composed  of  people 
from  bofli  the  scientific  community  and  the  public.  It  meets  three  or  four  times  a  year  to  review  grant  applications 
and  to  make  recommendations  on  matters  of  policy  and  research  emphasis.  In  addition  to  the  council,  the  director 
also  consults  with  intramural  investigators  ,scientists  in  the  extramural  program,  with  groups  of  patients  and  their 
ftimilies  interested  in  research  on  particular  diseases,  with  professional  and  scientific  groups,  with  representatives  of 
the  Administration  and  members  of  Congress,  and  with  the  public. 

In  1997,  the  NIH  published  “Setting  Research  Priorities  at  the  National  Institutes  of  Health”.  In  this 
booklet,  the  NIH  lists  five  criteria  on  which  the  Institutes  base  their  allocation  of  resources.  These  criteria  include 
the  heal*  needs  of  the  public,  the  scientific  quality  of  the  research,  the  potential  for  scientific  progress,  adequate 
support  of  infrastructure  (human  capital,  equipment  and  instrumentation,  and  facilities)  and  a  need  to  diversify  the 


*  The  OSS  intends  to  accomplish  the  scientific  investigations  related  to  Enterprise  and  Science  goals  within  a  five  to  six 
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research  portfolio  along  the  broad  and  expanding  frontiers  of  research.  Within  each  categoiy  listed  above  there  are 
criteria  that  help  each  manager  decide  on  the  best  course  for  funding.  For  example,  when  setting  criteria  for  the 
health  needs  of  the  public,  the  NIH  looks  at  a  combination  of  factors,  such  as  the  number  of  people  who  have  a 
particular  disease,  the  number  of  deaths  caused  by  a  disease,  the  degree  of  disability  produced  by  a  disease,  the 
degree  to  which  a  disease  cuts  short  a  normal,  productive,  comfortable  lifetime,  the  economic  and  social  costs  of  a 
disease,  and  the  need  to  act  rapidly  to  control  the  spread  of  a  disease. 

When  institute  staff  notice  evidence  of  an  emerging  area  of  research,  “they  assess  the  importance  of  the 
new  field  and  gauge  interest  and  capabilities.  They  can  then  convene  a  meeting  or  workshop,  write  up  a  proposal 
for  a  new  program,  and  go  to  their  council  for  approval.  If  the  program  does  not  have  a  known  constituency,  an 
institute  will  often  issue  a  request  for  applications.”  (OTA,  191;  p.  100) 

Despite  the  existence  of  these  internal  mechanisms  for  its  priority  setting  process,  NIH  has  shown  concern 
over  methodology  and  an  interest  in  receiving  external  validation  of  its  process.  NIH  contracted  the  National 
Research  Coimcil  to  look  into  four  issues  related  to  setting  priorities:  allocation  criteria,  the  decision-making 
process,  mechanisms  for  public  input,  and  the  impact  of  congressional  directives.  The  resulting  report.  Scientific 
Opportunities  and  Public  Needs  (NAS,  1998,)  recommended  that  in  setting  priorities  the  NIH  should  continue  to  use 
its  current  criteria.  Nevertheless,  it  should  also  try  to  create  a  better  public  understanding  of  its  priority  process  and 
the  NIH  should  be  more  systematic  and  use  more  formal  mechanisms  to  obtain  and  analyze  the  data  used  in  priority 
setting.  In  regards  to  the  actual  priority  setting  process,  the  committee  recommended  allocating  greater  power  to 
both  the  NIH  director  and  his  advisory  committee.  The  NRC  stated  that  directors  of  all  of  the  institutes  and  centers 
supply  multiyear  strategic  plans,  including  budget  scenarios,  to  the  NIH  director;  and  that  there  should  be  a  greater 
involvement  of  the  Advisory  Committee  in  the  priority-setting  process. 

The  National  Science  Foundation 

The  primary  role  of  NSF  is  to  support  basic  research  and  science  education  across  broad  categories  of 
science  and  engineering.  This  is  done  primarily  through  support  for  university-based  individual  investigators.  The 
NSF  long  range  planning  cycle  is  part  of  and  preparatory  to  preparing  the  budget  in  each  year.  “Divisions  and 
Directorates  consider  scientific  opportunities,  issues  and  constraints  confronting  them  as  well  as  national  needs  and 
trends  with  the  object  of  articulating  priorities  and  formulating  new  initiatives”  (NSTC,  1998).  The  efforts  are 
intended  to  track  into  four  core  strategies  for  achieving  broad  agency  mission  goals: 

•  develop  intellectual  capital 

•  strengthen  the  physical  infrastructure 

•  integrate  research  and  education 

•  promote  partnerships. 


year  time  frame. 
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The  NSF  process  for  strategic  planning  involves  calling  in  NSF  advisory  committees  of  visitors,  regular 
reviews  of  programs  and  input  from  the  National  Science  Board,  and  at  the  Directorate  level  reports  from  external 
groups  on  program  issues  (NSF,  1997).  (NSF,  perhaps  owing  to  the  specialized  nature  of  its  mission,  is  the  only 
mission  agency  whose  plans  explicitly  consider  integration  with  those  of  others.  This  is  principally  done  through 
personal  interaction  at  the  program  level  between  NSF  portfolio  managers  and  their  opposite  numbers.) 

Further,  OTA  describes  the  process  within  NSF  as  “very  much  a  bottom-up  process.  Goals  are  set  by 
scientific  opportunity  and  the  proposal  process,  as  well  as  in  special  initiatives  from  advisory  panels.”  OTA 
provides  fiirther  illustration: 

The  agency  primarily  sets  priorities  and  plans  through  a  process  described  by  many  as 
"...continuous,  open,  and  decentralized.”  The  decision  cycle  is  keyed  to  the  annual  Federal  budget  and 
annual  appropriation  cycles.  Eight  populations  provide  formal  and  informal  input  into  the  plarming 
process.  Ihey  are:  1)  the  National  Science  Board;  2)  advisory  committees;  3)  professional  societies;  4) 
NRC;  5)  Visiting  Scientists,  Engineers,  and  Educators  (also  known  as  “rotators’  ‘);  6)  NSF  staff;  7)  the 
Inter-Directorate  Task  Force;  and  8)  Congress.  (OTA,  1991;  p.  1 14) 

Setting  NSF  apart  from  other  agencies  is  the  role  of  die  National  Science  Board  (NSB)  in  priority  setting. 
According  to  OTA,  decisions  made  within  the  agency  are  forwarded  to  the  NSB  for  consideration  in  the  strategic 
plan. 


A  strategic  plan  is  developed  that  must  be  set  against  the  general  recommendations  of  NSB.  For 
example,  in  1989,  NSB  decided  on  four  general  priorities  for  NSF  to  pursue:  international  cooperation  in 
research,  education,  economic  competitiveness,  and  better  methods  for  leveraging  Federal  dollars  (i.e.,  to 
share  funding  with  other-typically  State  or  private-sources).  (OTA,  1991;  p.  115) 


“The  National  Science  Board  (NSB)  is  specifically  charged  with  assessing  the  health  of  science  in  the 
Nation  and  with  advising  the  President  and  Congress  on  matters  of  national  science  policy.”  A  particular 
responsibility  of  the  Board  in  implementing  this  mandate  is  the  biennial  publication  of  Science  and  Engineering 
Indicators,  [cited  in  the  National  Science  Foundation  Act  of  1950,  as  amended,  42  U.S.C.  Sec.  1 861  .et.  seq]  (NSB, 

1997) 

The  NSB  recommended  that  the  following  two  criteria  be  adopted  in  place  of  the  four  criteria  that  had  been 
used  in  the  past  to  determine  research  priorities:  What  is  the  intellectual  merit  of  the  proposed  activity?  E.g.,  does  it 
advance  knowledge  and  understanding  in  its  own  field  and  across  fields?  Is  it  creative  and  original?  What  are  the 
broader  impacts  of  the  proposed  activity?  E.g.,  advance  discovery  and  promote  teaching?  Enhancing  partnerships? 
This  criterion  relates  to  the  potential  of  the  proposed  research  to  contribute  to  better  understanding  or  improvement 
of  the  quality,  distribution,  or  effectiveness  of  the  Nation’s  scientific  and  engineering  research,  education,  and 
manpower  base.”  NSF  has  also  cited  the  following  criterion: 
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... questions  relating  to  scientific,  engineering,  and  education  personnel,  including  participation  of 
women,  minorities,  and  disabled  individuals;  the  distribution  of  resources  with  respect  to  institutions  and 
geographical  area;  stimulation  of  high  quality  activities  in  important  but  underdeveloped  fields;  support  of 
research  initiation  for  investigators  without  previous  Federal  research  support  as  a  Principal  Investigator  or 
Co-Principal  Investigator  and  interdisciplinary  approaches  to  research  or  education  in  appropriate  areas. 
(OTA) 

United  States  Environmental  Protection  Agency  (EPA) 

The  U.S.  EPA’s  mission  is  to  protect  human  health  and  to  safeguard  the  natural  environment-air,  water, 
and  land-upon  which  life  depends.  Within  the  Agency,  EPA’s  Office  of  Research  and  Development  (ORD)  has  the 
principal  responsibility  for  research  and  development.  According  to  the  National  Academy  of  Public  Administration 
(NAPA)  report,  "Setting  Priorities,  Getting  Results:  A  New  Direction  for  EPA  «  qRD’s  1 995  Strategic  Plan  relied  heavily  on 
EPA’s  Science  Advisory  Board  (an  independent  group  of  engineering  and  science  advisors  to  EPA)  and  expert 
panels  convened  by  NAPA  and  the  National  Academy  of  Sciences  (NAPA  ;NAS,  1995).  In  late  1995,  ORD 
distributed  a  review  draft  of  the  strategic  plan  to  solicit  comments  from  external  partners  and  stakeholders.  Based 
on  peer  reviewer  comments,  ORD  refined  and  expanded  the  strategic  plan  to  produce  the  final  version,  ORD ’s 
Strategic  Plan  (EPA,  May  1996). 

The  most  important  of  EPA’s  strategic  principles  is  the  explicit  use  of  the  risk  paradigm  to  shape  and  focus 
EPA’s  organizational  structure  and  research  agenda.  Risk  assessment  is  the  process  that  scientists  use  to  understand 
and  evaluate  the  magnitude  and  probability  of  risk  posed  to  human  health  and  ecosystems  by  environmental 
stressors,  such  as  pollution  or  habitat  loss  or  change*^.  The  risk  assessment  process  consists  of  four  steps:  hazard 
identification,  dose  response  assessment,  exposure  assessment,  and  risk  characterization.  Risk  management  options 
may  include  both  regulatory  programs  and  voluntary  activities  (e.g.,  recycling)  to  reduce  or  eliminate  production  of 
the  stressor.  The  risk  assessment  paradigm  has  been  defined  many  times  over  the  years,  most  notably  by  die  NAS, 
which  consolidated  and  gave  context  to  terms  that  had  been  defined  in  different  ways  up  to  that  point  (NAS,  1983; 
NRC,  1994;  NRC,  1996).  The  general  principles  set  forth  in  the  NAS  paradigm  are  still  useful  as  an  organizing 
focus  for  ORD’s  strategic  thinking.  ORD’s  commitment  to  develop  a  risk-based  research  agenda  has  required  a 
risk-based  process  for  selecting  and  ranking  those  research  areas  of  primary  importance  to  EPA.  ORD  provides 
suggestions  for  improving  science  at  the  EPA  in  order  to  institute  a  more  effective,  risk-based  research  program. 

For  example,  ORD  reorganized  their  nationwide  system  of  laboratories  to  conform  to  the  fimdamental  components 
of  the  widely  used  risk  assessment  and  risk  management  processes.  With  its  Strategic  Plan,  ORD  instituted  a  new 
system  for  determining  research  priorities  based  on  risk  assessment  and  risk  management  principles. 


I'l  Davies  was  at  the  forefront  of  the  use  and  potential  use  of  comparative  risk  for  priority  setting.  Davies  noted  that 
while  there  is  a  need  for  priority  setting  using  risk  comparisons,  the  limitations  of  the  tool  must  also  be  considered  (Davies,  1996) 
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EPA’s  strategic  plan  promotes  greater  partnership  between  ORD  and  its  primary  clients — ^EPA’s 
Program  and  Regional  Offices  and  the  external  scientific  community—  by  providing  clear  mechanisms 
and  opportunities  for  stakeholder  involvement.  EPA’s  strategic  plan  includes  a  detailed  description  of 
how  ORD  translates  its  research  priorities  into  a  research  program,  as  well  as  a  discussion  of  EPA’s 
commitment  to  providing  the  infrastructure  needed  to  conduct  high  quality  research. 

Department  of  Commerce  -  National  Institute  for  Standards  and  Technology 

According  to  agency  documents  and  Congressional  testimony,  the  National  Institute  of  Standards  and 
Technology  (NIST)  sets  priorities  in  specific  measurement  areas  based  on  the  advice  of  coxmcils  created  by  NIST 
itself  but  which  are  established  as  independent  nonprofit  organizations.  For  example,  NIST  sets  priorities  in  optical 
radiation  and  ionizing  radiation  measurement  based  on  the  advice  of  two  councils  created  by  NIST  and  established 
as  nonprofit  organizations.  The  Council  for  Optical  Radiation  Measurements  (CORM)  was  established  in  1972  to 
determine  ^'Pressing  Problems  and  Projected  National  Needs  in  Optical  Radiation  Measurement  .  CORM  is 
composed  of  members  from  government,  industry  and  academia.  The  Council  strives  to  provide  a  consensus  on 
industrial  and  academic  requirements  for  standards  and  programs,  including  setting  priorities.  CORM  was  so 
successful  in  setting  priorities  that  NIST  established  a  Council  for  Ionizing  Radiation  Measurements  and  Standards 
(CIRMS).  However,  NIST  also  uses  a  very  different  approach  to  priority  setting  for  its  service-oriented  division, 
Time  and  Frequency.  This  division  maintains  direct  contact  with  customers  and  manufacturers.  This  one-on-one 
contact  in  addition  to  periodic  customer  surveys  has  allowed  NIST  to  set  priorities  based  on  customer  need.  By 
working  with  its  customers,  NIST  is  able  to  answer  what  it  feels  is  its  fundamental  question  for  priority  setting,  “Do 
our  research  programs  meet  the  present  and  future  measurement  needs  of  the  United  States?”  (Gebbie,  1996). 
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Clinton,  William  J.,  and  Albert  Gore,  Jr.,  Science  in  the  National  Interest^  Washington,  D.C.:  Executive  Office  of 
the  President,  Office  of  Science  and  Technology  Policy,  August  1994. 

In  laying  out  the  foundation  for  Clinton  administration’s  support  for  science,  this  document  lists 
important  national  goals  to  which  science  can  contribute:  health,  prosperity,  national  security, 
environmental  responsibility,  and  improved  quality  of  life.  These  goals  should  be  met  by 
maintaining  leadership  across  the  frontiers  of  scientific  knowledge,  enhancing  connections  between 
basic  research  and  national  goals,  stimulating  public-private  partnerships,  ensuring  excellence  in 
science  education,  and  raising  scientific  literacy. 

Congressional  Research  Service,  The  Committee  for  the  National  Institute  for  the  Environment.  Research  and 
Development:  Priority  Setting  and  Consolidation  in  Science  Budgeting  IL  Issue  Brief.  Washington,  D.C.: 
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—  Analysis  of  Ten  Selected  Science  and  Technology  Policy  Studies,  Science  Policy  Research  Division,  Library  of 
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CRS  Report  97-70,  CRS  Report  97-382,  CRS  Report  RS20257,  and  CRS  Report  98-224. 

The  CRS  Reports  noted  that  the  House  Science  Committee  held  hearings  in  1996  and  1997  on 
implementation  in  the  civilian  science  agencies,  and  in  1997,  the  House  announced  this  as  a  major 
oversight  target.  Several  committees  asked  agencies  to  link  FY1999  and  FY2000  budget  requests  to 
goals  expressed  in  their  strategic  and  performance  plans. 

Cook-Deegan,  Robert  Mullan,  “Previous  Analysis  of  the  U.S.  R&D  Allocation  Process,”  working  paper, 
Washington,  D.C.:  National  Academy  of  Sciences,  January  1995. 

As  input  to  a  National  Academy  of  Sciences  effort  to  examine  allocation  of  the  federal  R&D  budget, 
this  report  summarizes  many  of  the  other  reports  included  in  this  bibliography. 

Committee  for  Economic  Development,  Americans  Basic  Research:  Prosperity  Through  Discovery.  New  York,  New 
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Council  on  Competitiveness,  Endless  Frontier,  Limited  Resources:  U.S.  R&D  Policy  for  Competitiveness, 
Washington,  D.C.:  Council  on  Competitiveness,  1996. 

This  report  addresses  the  ways  in  which  U.S.  industry,  government,  and  academia  have  been 
responding  to  the  R&D  environment  since  the  end  of  the  Cold  War.  The  council’s  central  finding  is 
that  R&D  partnerships  hold  the  key  to  America’s  future.  The  report  examines  the  role  of  partnerships 
in  R&D  innovations  in  the  aircraft,  automotive,  chemical,  electronics,  information  technologies,  and 
pharmaceuticals  sectors.  Finally,  the  report  provides  some  policy  guidelines  for  the  nation,  for 
industry,  for  government,  and  for  academia.  Specifically,  the  council  states  that  die  distinctions 
between  basic  and  applied  research  need  to  be  reevaluated  to  account  for  R&D  risks  in  the  short, 
medium,  and  long  terms.  It  calls  for  government  and  industry  to  work  together  in  choosing  future 
R&D  investments.  The  report  also  states  that  federal  research  should  reflect  today’s  missions  and 
budget  envirorunent. 
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Future”  Washington,  D.C.  1996. 

Davey,  Michael  E.,  Genevieve  J.  Knezo,  Richard  E.  Rowberg,  and  Wendy  H.  Schacht,  “Exploring  Alternative 
Models  of  Federal  Support  for  Research  and  Development,”  prepared  for  a  Congressional  Research  Service 
Workshop,  Washington,  D.C.:  The  Libraiy  of  Congress,  October  25, 1995. 

This  paper  covers  the  history  of  federal  R&D  support  and  the  emergence  and  growth  of  industrial 
R&D.  The  authors  suggest  four  models  of  federal  support  for  R&D.  The  first  model  proposes  that 
federal  R&D  support  would  be  for  defense  purposes  only,  eliminating  federal  support  for  all  other 
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R&D  activities.  Private  industry  would  be  responsible  for  funding  all  R&D  not  related  to  defense.  In 
tiie  second  model,  federal  R&D  support  would  cover  defense,  enviroimental  clean-up,  space,  and 
public  health;  70  to  80  percent  of  Itods  would  be  for  defense  R&D  with  the  remainder  going  toward 
the  other  areas.  The  third  model  would  reduce  federal  R&D  funding  by  30  to  50  percent  and  would 
treat  universities  and  the  private  sector  as  equal  partners  in  R&D  enterprise.  In  the  fourth  model, 
federal  R&D  support  would  mainly  cover  basic  research,  applied  R&D  would  be  reduced  by  30 
percent. 
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Request  for  R&D  and  Long-Term  Outlook  for  Science,  March  28, 1996. 

This  analysis  presents  the  House  Democratic  view  of  the  1997  R&D  budget,  comparing  the 
Republican  and  Democratic  budget  eillocations.  It  provides  a  historical  graphical  overview  of  past 
R&D  budgets  and  breaks  down  the  allocations  by  federal  departments  and  agencies,  including  the 
NSF,  NASA,  NIST,  DOE,  NOAA,  NIH  and  EPA. 

Department  ofEnergy,  Office  of  Science.  Strateg/c  Washington,  D.C:  June  1999. 

This  companion  document  to  DOE’s  Strategic  Plan  defines  scientific  accomplishments  and  provides 
priorities  and  strategies  for  the  future,  including  how  the  Office  of  Science  generates  priorities  in 
science. 

Environmental  Protection  Agency,  Science  Advisory  Board.  Reducing  Rist  Setting  Priorities  and  Strettegies  for 
Environmental  Protection.  Washington,  DC:  U.S.  Environmental  Protection  Agency.  September  1990. 

This  report  examines  strategies  for  reducing  major  risks  to  the  environment  by  improving 
methodologies  for  assessing  and  comparing  risk  reduction  options  for  the  future. 

Environmental  Protection  Agency,  Office  of  Research  and  Development,  Strategic  Plan,  Washington,  DC:  U.S. 
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This  biennial  report  from  the  President  to  Congress  on  S&T  addresses  the  administration’s  portfolio 
for  S&T  in  security,  health,  environment  and  hirnian  resources.  The  report  address  (1)  continuing 
American  leadership  in  the  S&T  enterprise;  (2)  strengthening  science,  math,  and  engineering;  (3) 
providing  for  a  sound  fiscal  and  regulatory  environment  for  research;  and  (4)  retaining  a  long-term 
commitment  to  research,  education,  and  innovation. 

—  “Memorandum  for  the  Heads  of  Executive  Departments  and  Agencies:  FY1998  Research  and  Development 

(R&D)  Priorities,”  John  H.  Gibbons  and  Alice  Rivlin,  May  29,  1996. 
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(R&D)  Priorities,”  John  H.  Gibbons  and  Leon  E.  Panetta,  May  6, 1994. 
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Standards  and  Technology  Before  the  House  Committee  on  Science  Subcommittee  on  Technology.”  May  2, 
1996. 

In  her  testimony,  Katherine  Gebbie  provides  several  ways  in  which  NIST  sets  priorities  among  its 
R&D  investments.  Specifically,  she  provides  a  model  in  which  NIST  looks  to  two  nonprofit  councils, 
composed  of  government,  industry  and  academe,  to  advise  it  on  setting  priorities  while  also 
discussing  another  model  that  is  survey  driven  and  involves  customer  feedback. 
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91 


The  Research  Roundtable,  an  HHS-led  interagency  group,  developed  guidance  for  R&D  performance 
measures.  The  GAO  described  the  difficulties  of  developing  research  outcomes  measures. 


Glauthier,  T.  J.,  “Research  and  Development  Priorities  and  Reinventing  Government,”  AAAS  Science  and 
Technology  Policy  Yearbook  1995,  Washington  D.C.:  American  Association  for  the  Advancement  of  Science, 
1995,  pp.  57-62. 

This  article  examines  the  current  pressures  on  the  federal  budget  and  the  Clinton  administration’s 
efforts  toward  reinventing  government.  Specifically,  the  article  recommends  that  the  science 
community  (1)  identify  the  implications  and  priorities  of  all  programs  being  put  together,  which 
programs  work  and  which  do  not;  (2)  explain  these  programs  in  laymen’s  terms,  for  congressional 
and  public  consumption;  and  (3)  reinvent  programs  by  including  industry,  universities,  and 
government  in  the  decisioiunaking  process. 

Hertzfeld,  Heniy  R.  “An  Assessment  of  Studies  of  the  Contribution  of  Fundamental  Research,”  draft 
submitted  to  the  National  Science  Foundation,  Directorate  for  Science,  Technology,  and  International 
Affairs,  Division  of  Policy  Research  and  Analysis,  October,  1990.  [photocopy  dated  May  11,  1992] 


Hill,  Christopher  T.,  “On  the  Allocation  of  Federal  R&D  Funds,”  working  paper,  Fairfax,  Va.:  The  Institute  of 
Public  Policy,  George  Mason  University,  January  1995. 

This  paper  covers  a  broad  overview  of  present-day  allocation  of  federal  R&D  funds  among  national 
purposes,  performers,  and  federal  agencies.  It  describes  the  historical  processes  tiiat  have  helped 
shape  R&D  allocation.  R&D  spending  responds  to  national  crises;  political  and  budgetary 
imperatives;  and  occasionally,  to  determined  efforts  of  highly  committed  interests  pursuing  long-term 
visions  of  a  different  future.  Rarely  does  the  R&D  portfolio  reflect  a  carefully  considered  balance 
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among  national  needs,  and  it  does  not  appear  to  take  account  of  the  overall  vitality  of  the  national 
R&D  enterprise. 

Knezo  Genevieve  J.,  Research  and  Development  Funding  in  a  Constrained  Budget  Environment:  Alternative 
Support  Sources  and  Streamlined  Funding  Mechanisms,  report  for  Congress,  Washington,  D.C.:  Congressional 
Research  Service,  The  Library  of  Congress,  April  5, 1996. 

Tliis  report  summarizes  cuts  made  in  federal  R&D  funds,  proposals  to  increase  nonfederal  sources  of 
funding,  proposals  to  improve  R&D  priority-setting,  and  ways  to  streamline  and  alter  existing  R&D 
award  processes  to  make  them  more  efficient.  Specific  topics  include  limiting  federal  R&D  support 
to  targeted  areas;  creating  a  Department  of  Science  and  Technology  to  coordinate  federal  R&D 
funding-  expanding  support  to  such  alternatives  as  lotteries,  state  funding,  and  foundations;  revising 
the  tax  code  to  promote  R&D;  modifying  grant  mechanisms  to  conserve  resources;  and  eliminating 
conservation  of  peer-reviewed  science. 

_ Research  and  Development:  Priority  Setting  and  Consolidation  in  Science  Budgeting.  The  Cominittee  for  the 

National  Institute  for  the  Environment,  Research  and  Development:  Priority  Setting  and  Consolidation  in 
Science  Budgeting  II.  October,  1999. 

_ "Defense  Basic  Research  Priorities:  Funding  and  Policy  Issues,  CRS  Report  for  Congress ,  Washington,  DC. 

Congressional  Research  Service,  Oct.  24, 1990. 

Kostoff,  Ronald  N.  Handbook  of  Research  Impact  Assessment,  7th  Edition,  DTIC  Report  No.  ADA29602I, 
Summer  1997. 

Lederman,  Leon  M.,  Science:  The  End  of  the  Frontier?  Washington,  D.C.:  American  Association  for  the 
Advancement  of  Science,  January  1991. 

This  is  a  formal  survey  conducted  to  gauge  the  state  of  academic  research  in  U.S.  uiiiversities. 

University  faculty  were  asked  about  tiieir  experiences  with  research  funding.  Questions  were  ask^ 
about  research  funding  in  the  scientist’s  own  research  area,  the  relative  ease  or  difficulty  of  obtaining 
funding,  experiences  with  research  funding  for  the  future,  and  factors  influencing  the  ability  of 
scientists  to  conduct  research  in  their  current  settings.  The  results  show  that  academic  research  is  in 
trouble;  research  funding  is  diminishing;  and  morale  is  flagging.  According  to  the  AAAS,  university 
funding,  when  corrected  for  inflation,  was  only  slightly  higher  in  1990  than  it  was  in  1968.  The 
report  concludes  that,  to  balance  the  complexity  of  science,  the  United  States  should  be  spending 
twice  the  amount  it  was  investing  in  1968,  or  $10  billion/year,  ^d  this  investment  should  increase 
least  4  percent  per  year.  The  report  further  recommends  establishing  a  commission  to  deal  with  this 
issue,  consisting  of  representatives  from  the  Executive  and  Legislative  Brariches  of  the  federal 
government,  industry,  the  financial  community,  and  the  academic  commimity. 

Levine,  Candice  S.,  “Reorienting  for  Sustainable  Development:  Support  for  a  National  Science  and  Technology 
PoWcy,”  Journal  of  International  Affairs,  Spring  1998. 

This  article  discusses  R&D  funding  as  it  pertains  to  the  environment.  In  general,  the  articles  states 
that  poor  coordination  among  federal  agencies  and  inconsistent  support  from  national  leaders  result  in 
a  poor  national  science  policy  that  will  hinder  sustainable  development. 

McLoughlin,  Glenn  J.,  International  Science  and  Technology:  Issues  For  U.S.  Policymakers,  Report  for  Congress, 
Washington,  D.C.:  Congressional  Research  Service,  The  Library  of  Congress,  September  16,  1994. 

This  report  presents  a  comparison  of  national  R&D  funding  issues  and  options,  how  other  nations  set 
S&T  priorities,  fund  research  and  development  (R&D)  programs,  and  address  similar  S&T  issues. 

Each  nations'  R&D  funding  figures  are  presented  in  current  U.S.  dollars  in  several  major  R&D 
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categories,  Other  R&D  funding  categories  may  be  found  in  the  chapters  on  each  nation  (US, 

Germany,  Japan,  India,  Australia,  UK,  New  Zealand,  and  the  EU). 

—  International  Science  and  Technology  Issues:  Summary  of  a  Report  to  the  Committee  on  Science.  Report  to 

Congress,  Washington,  D.C.:  Congressional  Research  Service,  Science,  Technology,  and  Medicine  Division, 
April  20, 1998,  15p. 

This  report  is  a  summary  of  a  comparative  study  on  international  science  and  technology,  prepared  at 
the  request  of  the  Committee  on  Science  of  the  House  of  Represents.  It  provides  a  digest  of  analysis 
and  findings  on  the  science  and  technology  policies,  civilian  research  and  development  funding,  and 
relevant  policy  issues  of  13  countries  and  the  European  Union.  It  also  provides  a  description  of  why 
these  findings  and  issues  may  be  of  interest  to  U.S.  policymakers,  as  well  as  an  analysis  of  issues  and 
concerns  about  U.S.  data  collection  and  information. 

—  and  R.  E.  Rowberg,  Linkages  between  Federal  Research  and  Development  Funding  and  Economic  Growth.  A 

Report  in  the  Series  Economic  Policymaking  in  Congress:  Trends  and  Prospects.  Report  for  Congress, 
Washington,  D.C.:  Congressional  Research  Service,  The  Library  of  Congress,  1992. 19  p. 

This  report  provides  information  on  other  forms  of  "direct"  (R&D)  and  "indirect"  (other  areas)  S&T 
policies  in  Ae  United  States).  The  report  examines  the  contribution  of  Federal  R&D  to  economic 
growth.  It  addresses  the  question  about  a  more  direct  Federal  role  in  the  context  of  international 
economic  competition.  Finally,  policy  considerations  are  presented  which  could  affect  the 
relationship  between  mission  agency  R&D  and  economic  growth,  and  the  issues  surrounding  a  more 
direct  Federal  role. 

McGeaiy,  Michael,  “Where  Does  the  Federal  Dollar  for  Basic  Research  Go?”  working  paper,  submitted  to  the 
National  Academy  of  Sciences  Committee  on  Criteria  for  Federal  Support  of  Research  and  Development  meeting 
held  on  March  30-April  1, 1995,  in  Washington,  D.C. 

This  paper  provides  information  on  federally  funded  basic  research.  Using  NSF  1994  S&T  data,  the 
paper  discusses  who  provides  the  funds  for  basic  research  and  why,  describes  which  mechanisms  are 
used  to  provide  support,  and  discusses  which  agencies  and  departments  conduct  basic  research  and  in 
which  areas.  It  also  provides  a  general  overview  and  breakout  of  indirect  costs  paid  to  universities 
and  other  extramural  research  institutions. 

Merrill,  Stephen  A.,  and  Michael  McGeary,  “Balancing  the  Federal  Research  Portfolio:  Who’s  Deciding  and  Why?’ 
working  paper,  Washington,  D.C.:  National  Research  Council,  National  Academy  of  Sciences,  August  1 1, 1999. 

The  authors  take  issue  with  congressional  claims  that  the  current  R&D  budgeting  process  results  in  a 
well-balanced  portfolio.  By  examining  NSF  data,  the  paper  identifies  15  areas  of  R&D  in  which 
funds  have  declined  during  the  1990s.  The  authors  note  that  no  one  agency  is  responsible  for 
ensuring  that  this  drop  in  R&D  funding  is  not  harming  the  national  interest.  They  call  for  (1)  a 
bottom-up  evaluation  of  these  cuts,  (2)  a  more  open  discussion  of  national  S&T  priorities,  and  (3) 
principle  policymaking  bodies  to  make  adjustments  to  the  funding  portfolio  when  there  appears  to  be 
a  serious  shortfall  in  desirable  investment. 


National  Academy  of  Sciences.  National  Research  Council,  Evaluating  Federal  Research  Programs:  Research  and 
the  Government  Performance  and  Results  Act.  Washington,  D.C.:  National  Academy  Press,  February  1999. 

The  NRC  released  report  recommended  that  federal  agencies  develop  performance  measures  for  research, 
and  issued  "benchmarking"  reports  comparing  the  status  of  U.S.  science  to  other  countries  for  mathematics, 
materials  science  and  engineering,  and  immimology.  The  agencies  submitted  strategic  plans  to  the  Congress 
in  September  1997  and  delivered  annual  performance  plans  with  FY1999  budget  justifications. 

—  National  Research  Council,  Harnessing  Science  and  Technology  for  America 's  Economic  Future:  National  And 
Regional  Priorities.  Washington  DC:  National  Academy  Press,  1999. 
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Report  makes  recommendations  to  develop  new  mechanisms  for  international  research  collaboration 
to  advance  fundamental  knowledge,  drawing  on  the  experience  of  recent  years.  After  the  federal 
government,  the  academic  institutions  performing  research  and  development  (R&D)  provided  the 
second  largest  share  of  academic  R&D  support.  The  NRC  report  noted  that  much  of  this  funding 
comes  from  state  governments,  but  is  counted  as  institutional  funding  because  die  university  has 
discretion  over  whether  it  will  be  spent  on  research  or  in  other  ways.  Industrial  R&D  support  for 
academic  institutions  has  grown  more  rapidly  than  support  from  other  sources  since  1980  (i.e.,  in 
constant  dollars,  industrial-financed  R&D  increased  by  an  estimated  250%  from  1980  to  1995,  and 
industry’s  share  grew  from  3.9%  to  6.9%)  (NRC,  1999).  More  extensive  university-industry 
collaboration  on  long-term  issues  of  interest  to  industry  could  help  to  alleviate  the  funding  pressures 
being  faced  by  universities  (NRC,  1999). 

National  Research  Council.  An  Assessment  of  the  National  Institute  of  Standards  and  Technology  Measurement 
and  Standards  Laboratories.  Washington  DC;  National  Academy  Press,  1998. 

Each  year  since  1959,  the  National  Research  Council  has  assessed  the  programs  of  the  National 
Institute  of  Standards  and  Technology  (NIST),  and  its  predecessor,  the  National  Bureau  of  Standards. 
Assessments  are  currently  performed  by  about  150  leading  scientists  and  engineers,  equally  from 
U.S.  industry  and  academe,  appointed  by  the  National  Research  Council  (NRC),  and  administered  by 
the  NRC's  Board  on  Assessment  of  NIST  Programs.  There  are  currently  seven  major  Panels  that 
assess  the  major  oiganizational  areas:  electronics  and  electrical  engineering,  manufacturing 
engineering,  chemical  science  and  technology,  physics,  materials  science  and  engineering,  building 
and  fire  research,  and  information  technology. 


National  Research  Council.  “International  Benchmarking  of  US  Materials  Science  and  Engineering  Research,” 
Committee  on  Science,  Engineering  and  Public  Policy.  Washington,  DC:  National  Academies  Press,  1998. 

Observations  on  the  President's  Fiscal  Year  1999  Federal  Science  and  Technology  Budget,  Washington, 

DC;  National  Academies  Press,  April  1998. 

Institute  of  Medicine,  Scientific  Opportunities  and  Public  Needs:  Improving  Priority  Setting  and  Public  Input  at 
the  National  Institute  of  Health,  Washington,  D.C.:  National  Academy  Press,  1998. 

This  report  examines  the  way  in  which  NIH  sets  priorities  and  provides  a  few  recommendations  for 
improvement.  The  report  states  that  NIH’s  objectives  should  revolve  around  identifying  the  public’s 
health  needs,  extending  basic  research  and.  The  report  recommends  that  NIH  continue  to  use  its 
current  method  for  criteria  setting,  but  implement  a  more  systematic  use  and  analysis  of  data  sources 
for  input  in  priority  setting  .  The  report  also  reconunends  an  increased  role  for  NIH’s  Advisoiy 
Committee  as  well  as  the  establishment  of  a  Public  Liaison  Office. 


National  Research  Council,  A  New  Science  Strategy  for  Space  Astronomy  and  Astrophysics,  Washington,  D.C.; 
National  Academy  Press,  1997. 

This  update  to  the  1991  report.  The  Decade  of  Discovery  in  Astronomy  and  Astrophysics,  uses 
priority-setting  methods  established  in  1991  to  provide  a  strategy  for  space  astronomy  and 
astrophysics.  In  doing  its  priority  setting,  the  Task  Group  on  Space  Astronomy  and  Astrophysics,  the 
community  (1)  concentrated  on  the  scientific  objectives  rather  than  the  method;  (2)  prioritized 
scientific  questions  according  to  whole  classes  of  astronomical  objects,  rather  than  to  individual 
observing  bands;  and  (3)  looked  realistically  at  cost  and  technical  feasibility. 

National  Research  Council.  Understanding  Risk:  Irtforming  Decisions  in  a  Democratic  Society.  Washington, 
D.C.  National  Academy  Press,  1996. 
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—  Board  on  Environmental  Studies  and  Toxicology.  “Interim  Report  of  the  Committee  on  Research  and  Peer 
Review  at  EPA.”  Washington,  D.C.  National  Academy  Press,  1995. 

—  Committee  on  Criteria  for  Federal  Support  of  Research  and  Development,  Allocating  Federal  Funds  for  Science 
and  Technology,  Washington,  D.C.:  National  Academy  Press,  1995. 

The  Committee  on  Criteria  for  Federal  Support  of  R&D  provides  an  overview  of  how  R&D  is 
defined  within  the  federal  government  and  describes  the  current  process  of  allocating  R&D  funds 
through  federal  departments  and  agencies.  Based  on  this  information  and  a  literature  review,  the 
conunittee  recommends  three  policy  initiatives  for  allocating  federal  funds:  (1)  The  President  should 
present  an  annual  comprehensive  FS&T  budget;  (2)  the  departments  and  agencies  should  make  FS&T 
allocation  decisions  based  on  clearly  articulated  criteria  that  are  congruent  with  those  that  the 
Executive  Office  of  the  President  and  Congress  use;  and  (3)  Congress  should  create  a  process  that 
examines  the  entire  Federal  Science  &  Technology  budget  before  the  total  federal  budget  is 
disaggregated. 

—  National  Research  Council.  Science  and  Judgement  in  Risk  Assessment.  Washington,  D.C.:  National  Academy 

Press,  1994. 

—  Science,  Technology,  and  the  Federal  Government:  National  Goals  for  a  New  Era.  Washington  D.C.:  National 
Academy  Press,  1993. 

This  report  recommends  tying  S&T  goals  to  two  overarching  principles:  (1)  The  U.S.  should  be 
among  the  world  leaders  in  all  major  areas  of  S&T,  and  (2)  die  U.S.  should  maintain  clear  dominance 
in  scientific  fields  likely  to  contribute  to  substantially  important  economic,  social,  or  cultural 
objectives.  Further,  government  should  cooperate  with  the  private  sector  to  maintain  U.S.  leadership 
in  technologies  that  promise  to  have  major  influence  on  industrial  and  economic  performance  and 
that  could  lead  to  new  industries,  based  on  principles  of  cost-sharing,  insulation  from  distributional 
politics,  and  stable  support. 


—  National  Research  Council,  Astronomy  and  Astrophysics  Survey  Committee,  The  Decade  of  Discovery  in 
Astronomy  and  Astrophysics.  Washington,  D.C.:  National  Academy  Press,  1991. 

ITiis  report  discusses  the  results  of  a  survey  conducted  by  the  Astronomy  and  Astrophysics  Survey 
Committee  of  the  NRC.  The  study  was  commissioned  to  provide  an  overview  of  what  is  going  on  in 
astronomy  and  to  recommend  initiatives  for  the  coming  decade.  The  committee  was  tasked  to 
provide  a  prioritized  list  of  instruments  for  the  coming  decade,  evaluate  the  existing  infrastructure, 
explore  the  consequences  of  the  computer  revolution  for  astronomy,  prepare  a  popular  summary  of 
opportunities  for  scientific  advances  in  astronomy,  and  suggest  possible  areas  for  developing  new 
observational  technologies. 

—  Presidents  of  the  National  Academy  of  Sciences,  National  Academy  of  Engineering,  and  Institute  of  Medicine, 
Federal  Science  and  Technology  Budget  Priorities.  Washington,  D.C.:  National  Academy  Press,  1988. 

This  report  notes  that  the  absence  of  a  coordinated  national  R&D  budget  and  lack  of  suitable  criteria 
for  maWng  global  R&D  budget  decisions  hinders  effective  use  of  federal  dollars.  The  report  notes 
that  priority-setting  within  agency  missions  is  adequate  and  that  a  pluralistic  approach  to  budgeting 
has  been  a  strength  of  the  U.S.  system.  In  three  classes  of  activity,  however,  special  attention  is 
needed:  (1)  initiatives  contributing  to  the  science  base,  (2)  initiatives  tied  to  presidential  or 
congressional  directives,  and  (3)  major  “megascience”  projects  slated  for  rapid  growth  or  large  pieces 
of  the  budget. 

— Risk  Assessment  in  the  Federal  Government:  Managing  the  Process.  Washington,  D.C.:  National  Academy  Press, 
1983. 
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National  Institutes  of  Health.  Setting  Priorities  at  the  National  Institutes  of  Health .  Washington,  D.C.:  Department 
of  Health  and  Human  Services,  September  1997. 


NIH’s  Working  Group  on  Priority  Setting  provides  a  description  of  the  way  that  the  NIH  set’s 
priorities.  According  to  the  Group,  the  NIH  provides  funding  to  programs  by  1)  responding  to  public 
health  needs  2)  following  a  stringent  peer  review  system  and  3)  diversifying  its  research  portfolio  to 
include  a  variety  of  research.  Input  into  which  research  programs  NIH  will  pursue  depends  on  a  the 
advise  of  a  variety  of  actors  from  the  extramural  science  community  to  Congress  and  the 
Administration. 

National  Institutes  of  Health.  NCRR  A  Catalyst  for  Discovery.  A  Plan  for  the  National  Center  for  Research 
Resources  1998-2003.  Bethesda,  MD:  Office  of  Science  Policy,  NCRR/NIH,  1998. 

National  Science  Board,  “Government  Funding  of  Scientific  Research,”  working  paper,  Washington,  D.C.:  U.S. 
Government  Printing  Office,  NSB-97-186, 1997. 

The  NSB  report  calls  for  mandatory  priority  setting  and  coordination  of  federal  R&D.  Report 
provides  a  follow  up  to  its  1997  announcement  that  NSB  would  play  a  larger  role  in  setting  national 
S&T  priorities  and  policy.  Separate  House  and  Senate  science  policy  efforts  are  also  described  under 
the  FY1999  budget  section. 

—  “Overview:  Science  and  Technology  in  Transition  to  the  21st  Century,”  in  Science  and  Engineering  IndicaXacs, 

Washington,  D.C.:  National  Science  Foimdation,  1998. 

National  Aeronautics  and  Space  Administration.  “NASA  Performance  Plan  Fiscal  Year  1999.”  Washington,  DC: 
U.S.  Government  Printing  Office,  February  1999. 

This  GPRA  document  describes  performance  measures  for  FY99  Budget  activities. 

National  Science  Foundation.  “FY1999  GPRA  Performance  Plan”,  Washington,  DC:  National  Science  Foundation, 
January  1999. 

—  GPRA  Strategic  Plan  FY1997-FY2003.  Washington,  DC:  National  Science  Foundation,  September  1997. 

National  Science  and  Technology  Council.  National  Nanotechnology  Initiative:  Leading  to  the  Next  Industrial 
Revolution.  February  2000. 

~  Strategic  Planning  Document  -  Transportation  R&D.  National  Science  and  Technology  Council  (NSTC) 
1995  Strategic  Planning  Documents,  March,  1995. 

This  summary  report  of  Federal  transportation  research  and  development  priorities  was  prepared  for 
die  National  Science  and  Technology  Council  (NSTC)  by  the  NSTC  Interagency  Coordinating 
Committee  on  Transportation  R&D  and  the  Office  of  Science  and  Technology  Policy.  The  strategic 
plan  reflects  die  initial  efforts  of  the  Committee  to  assess  Federal  research  and  to  develop  long-term 
R&D  programs  integrated  across  agencies  in  specific  transportation-related  areas  of  common  interest. 

It  is  based  primarily  on  materials  developed  by  the  subcommittees  and  working  groups,  working 
within  the  framework  established  by  the  full  committee  in  its  Strategic  Budget  Guidance  report 
presented  to  NSTC  in  April,  1994. 

The  summary  report  was  compiled  from  subcommittee  submissions  by  staff  of  DOTs  Volpe  National 
Transportation  Systems  Center  under  the  direction  of  Noah  Rifkin,  Executive  Secretary  of  the 
Committee  and  DOT  Director  of  Technology  Deployment  and  by  the  White  House  Office  of  Science 
and  Technology  Policy.  The  subcommittee  report  contains  extensive  additional  detail  concerning 
agency  programs,  goals,  issues  and  resources.  Efforts  of  the  Committee  in  1994,  summarized  in  this 
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document,  focused  on  identification  of  perceived  R&D  gaps  and  opportunities.  They  provide  the 
foundation  for  generation  in  1995  of  a  detailed  and  comprehensive  description  of  Federal 
transportation  R&D  goals,  plans,  measures,  budgets  and  priorities,  including  active  coordination  with 
odier  NSTC  Committees. 

— 1998  Annual  Report,  Washington,  D.C.:  National  Science  and  Technology  Council  Program  Office,  1998. 
—1997  Annual  Report,  Washington,  D.C.:  National  Science  and  Technology  Council  Program  Office,  1997. 

Office  of  Technology  Assessment,  Federally  Funded  Research:  Decisions  for  a  Decade,  Washington,  D.C.:  U.S. 
Government  Printing  Office,  OTA-SET-490,  May  1991. 

This  study  suggests  that  the  criteria  used  to  set  priorities  for  various  areas  of  research  lack  explicit 
guidelines,  particularly  at  the  highest  levels  of  allocation,  leading  to  widely  varying  criteria  and 
outcomes.  OTA  also  commented  that  the  lack  of  a  mechanism  for  evaluating  the  total  research 
portfolio  of  die  federal  government  in  terms  of  progress  toward  many  national  objectives  results  in 
S&T  being  only  loosely  tied  to  needs.  Finally,  the  federal  S&T  enterprise  should  seek  to  include 
criteria  beyond  scientific  merit  and  mission  relevance  when  judging  the  worth  of  a  research  program. 

The  report  calls  for  OSTP  to  disclose  the  criteria  by  which  federal  S&T  priorities  are  set. 

President’s  Committee  of  Advisors  on  Science  and  Technology,  Review  of  the  Proposed  National  Nanotechnology 
Initiative,  November  1999. 

—  Letter  to  President,  6  December  1996. 

—  PCAST  Fusion  Review  Panel,  The  U.S.  Program  of  Fusion  Energy  Research  and  Development,  1 1  July  1995. 

Popper,  Steven  W.,  “Policy  Perspectives  on  Measuring  the  Economic  and  Social  Benefits  of  Fundamental  Science,” 
RAND  MR-1 130-STPI.  Santa  Monica,  CA:  RAND,  September  1999. 

Press,  Frank,  “Criteria  for  the  Choice  of  Federal  Support,”  AAAS  Science  and  Technology  Policy  Yearbook 
1996/1997,  Washington  D.C.:  American  Association  for  the  Advancement  of  Science,  1997,  pp.  171-178. 

Frank  Press,  chair  of  the  committee  that  published  a  report  eaUtled  Allocating  Federal  Funds  for 
Science  and  Technology,  builds  upon  and  reacts  to  ideas  put  forth  in  the  report.  The  articles  states 
that  we  need  to  make  the  idea  of  a  federal  science  and  technology  budget  a  reality,  one  that  not  only 
contains  budget  numbers  and  definitions  but  also  provides  a  process  for  upgrading  the  S&T  portfolios 
of  agencies  by  forcing  trade-offs.  Appropriations  for  this  budget  can  be  debated  within  a  new 
subcommittee  created  for  the  specific  purpose  of  evaluating  the  FS&T  budget.  This  type  of  structure, 
however,  has  come  under  criticism  for  several  reasons  (1)  it  might  make  the  FS&T  budget  vulnerable 
during  times  of  budget  deficits,  (2)  it  may  result  in  a  decrease  in  the  overall  budget  pool  for  S&T,  and 
(3)  it  may  create  conflict  within  the  science  community  to  increase  the  budget  instead  of  complying 
with  constrictions,  and  (4)  because  the  NAS  report  is  itself  viewed  biased  in  favor  of  federal  labs  and 
universities. 


Robinson,  David  Z.,  “Think  Twice  before  Overhauling  Federal  Budgeting,”^Ay4S  Science  and  Technology  Policy 
Yearbook  1996/1997,  Washington  D.C.:  American  Association  for  the  Advancement  of  Science,  1997,  pp. 
217-224. 

This  article  disagrees  with  the  ideas  proposed  in  the  NAS  report.  Allocating  Federal  Funds  for 
Science  and  Technology,  calling  an  FS&T  budget  structure  conceptually  and  practically  "wrong. 

Robinson  continues  to  state  that  trade-offs  should  not  be  made  between  categories  of  FS&T 
investment  but  between  S&T  and  other  expenditures  within  a  federal  agency.  He  continues  to  argue 
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fliat  mechanisms  are  already  in  place  to  review  specific  areas  of  R&D  duplication.  Robinson 

recommends  tiiat,  instead,  policymakers  should  determine  the  appropriate  level  of  support  by  linking 
FS&T  programs  to  national  goals  while  making  trade-offs  between  current  and  future  needs. 


Saunders,  Kenneth  V.  et  al.,  Priority-Setting  and  Strategic  Sourcing  in  the  Naval  Research,  Development,  and 
Technology  Infrastructure.  Santa  Monica,  CA:  RAND,  1995. 


This  report  suggests  ways  in  which  the  Dept,  of  the  Navy  might  realize  more  value  fixim  its 
increasingly  constrained  research,  development,  and  technology  (RD&T)  dollars.  The  study  was 
motivated  by  the  Navy's  immediate  policy  needs  in  connection  with  the  1995  round  of  Base 
Realignment  and  Closure  (BRAC)  and  its  longer-term  need  to  make  the  best  use  of  its  resources. 

Suggestions  are  presented  in  three  parts.  First,  the  authors  develop  and  apply  a  framework  for  setting 
funding  priorities  in  the  Naval  RD&T  infrastructure.  Second,  the  authors  discuss  alternative  RD&T 
procurement  arrangements  that  are  seeing  increasing  use  in  the  private  sector  and  that  have  been  used 
in  various  parts  of  the  government.  These  are  commonly  called  "smart  buying,"  but  the  au&ore  use 
the  term"  strategic  sourcing."  Third,  the  authors  present  a  speculative  combination  of  the  priority¬ 
setting  and  strategic-sourcing  considerations  of  Ae  first  two  parts.  Using  a  reinterpretation  of  the 
orthogonal  plot  developed  earlier  in  the  report,  it  suggests  a  way  to  help  determine  which  parts  of  the 
Naval  RD&T  infrastructure  are  best  suited  for  alternative  procurement  arrangements.  It  also  suggests 
a  way  to  determine  which  facilities  might  be  involved. 

Science  and  Government  Report,  Mar.  1, 1997. 

The  Vice  President  for  Research  at  the  University  of  Michigan,  proposed  a  high-level 
public/govemmental  commission  to  assess  "'the  rationale  for  investments  in  research’ 
by. ..governments,  industry  and  universities...the  division  of  labor  among  academic,  industrial  and 
govermnent  laboratories;  criteria  for  setting  levels  of  R&D  support,  and  the  implications  of  current 
long-term  spending  projections  for  research." 

Shapley,  Willis  H.,  The  Budget  Process  and  R&D.  Washington,  D.C.:  Carnegie  Commission  on  Science, 
Technology,  and  Government,  April  1 992. 

This  report  describes  and  discusses  the  federal  budget  process  with  a  focus  on  R&D.  Shapley 
proposes  and  addresses  several  concerns  diat  have  arisen  in  creating  a  federal  R&D,  including  (1) 
setting  priorities  and  achieving  balance,  (2)  the  use  of  budget  data,  (3)  the  stability  and  continuity  of 
die  current  budget  process,  and  (4)  the  fragmentation  of  R&D  in  the  budget  review  process  in  OMB 
and  Congress.  Shapley  states  that  (1)  R&D  funding  for  programs  should  not  be  pitted  against  each 
other  but  rather  against  die  overall  federal  budget;  (2)  a  comprehensive  comparable  databank  on 
R&D  budgets  should  be  established,  as  proposed  in  a  1988  report  of  the  Senate  Budget  Committee; 

(3)  a  partial  rather  than  immediate  implementation  of  a  two-year  appropriation  cycle  is  more 
politically  saleable  and  (4)  subcommittee  hearings  for  R&D  should  not  be  done  by  a  separate 
committee,  because  this  could  make  R&D  agencies  more  vulnerable  to  arbitrary  reductions.  Finally, 

Shapley  states  that  there  is  a  significant  shortfall  in  the  R&D  budget  in  meeting  important  needs  and 
grasping  important  opportunities.  The  report  states  that  the  nature  of  R&D  makes  it  a  necessity  to 
increase  funding  in  certain  S&T  areas  to  keep  pace  with  advances;  however,  this  is  difficult  to 
accomplish  in  a  deficit-ridden  budget. 

Smith,  Philip  M.,  and  Michael  McGeary,  “Don’t  Look  Back;  Science  Funding  for  the  Future,”  Issues  in  Science  and 
Technology  Online,  Spring  1997. 

This  article  stresses  that  evolving  national  priorities  and  budget  constraints  call  for  a  new  approach  to 
federal  spending.  Corroborating  the  NAS  report.  Allocating  Federal  Funds  for  S&T,  the  article  calls 
for  the  development  and  use  of  a  federal  S&T  budget.  It  calls  on  the  OMB  and  OSTP  to  implement 
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an  annual  FS&T  analysis  as  a  part  of  the  normal  budget  review.  The  authors  state  that  using  this 
analysis  would  help  the  most  productive  programs  under  a  tight  budget  while  strengthening  the  case 
for  making  larger  investments  in  R&D. 


Teich,  Albert,  “Choosing  Among  Disciplines,”  AAAS  Science  and  Technology  Policy  Yearbook  1991,  Washington, 
D.C.:  American  Association  for  the  Advancement  of  Science,  1991,  pp.  41-45 

Teich  discusses  the  conflicting  perceptions  that  have  arisen  as  a  result  of  priority-setting  discussions. 

The  article  addresses  (1)  who  should  do  the  priority  setting,  (2)  who  would  use  the  results  and  how, 
and  (3)  what  the  outcomes  of  the  process  would  be.  The  articles  also  states  that  despite  the  concerns 
that  have  arisen,  the  budget  process  would  benefit  fi-om  the  change.  Furthermore,  incorporating 
priority-setting  methods  based  on  technological  merit,  scientific  merit  and  social  merit  would  greatly 
improve  the  process. 


The  Government  Performance  and  Results  Act,  P.L.  101-189  and  P.L.  100-456 

Require  the  Department  of  Defense  (DOD)  and  the  Office  of  Science  and  Technology  Policy  (OSTP) 
to  identify  priorities  for  critical  dual-use  technologies  for  national  security  and  economic  prosperity. 

Thomas,  Eleanor.  Strategic  Planning  at  the  National  Science  Foundation.  Arlington,  VA:  National  Science 
Foundation,  July  1996. 

U.S.  Congress,  The  Senate  FY1999  VA/HUD/Independent  Agencies  Appropriations  Report,  Washington,  DC: 
Congressional  Printing  Office,  1999. 

The  Senate  FY1999  report  called  on  NSF  to  identify  quantifiable  goals  for  research.  The 
2q)propriations  act,  P.L.  105-276,  gave  OSTP  and  0MB  authority  to  seek  the  NAS  study,  as  in  S. 

2217  (in  the  105th  Congress),  but  did  not  include  the  related  provisions. 

—  House  Majority  Leader  Report.  Washington,  DC:  Congressional  Printing  Office,  1999. 

The  House  Majority  Leader  issued  a  report  "rating”  the  FY1999  plans.  The  House  Committee  on 
Government  Reform  and  Oversight  and  the  Senate  Committee  on  Governmental  Affairs  held 
hearings  on  implementation. 

—  House  of  Representatives,  Committee  on  Science,  Unlocking  Our  Future:  Toward  a  New  National  Science 
Policy.  Washmgton,  DC:  Congressional  Printing  Office,  1998. 

Recognizing  that  choices  about  funding  R&D  must  be  made  in  the  face  of  limited  federal  resources, 
this  report  says  that  priorities  for  spending  on  science  and  engineering  will  have  to  be  set.  Because  of 
its  unique  role,  fundamental  research  in  a  broad  spectrum  of  scientific  disciplines,  administered 
through  the  peer  review  process,  should  receive  priority  for  federal  spending.  A  “sharp  eye”  should 
be  kept  on  possible  downstream  applications  for  such  research.  Mission-oriented  research  should 
continue  to  fund  highly  relevant,  noncommercial,  long-term  research. 


—  The  Government  Performance  and  Results  Act,  P.L.  103-62.  Washington,  DC:  U.S.  Congress  Printing  Office, 
1993. 

GRPA  requires  agencies  to  define  long-term  goals,  set  specific  annual  performance  targets,  and 
report  annually  on  performance.  Legislative  language  noted  the  difficulty  of  quantitatively  measuring 
some  program  outputs  and  allows  alternatives. 
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